HTTP Protocols and Requests

HTTP stands for Hypertext Transfer Protocol. HTTP is a stateless application layer protocol. Default ports used are 80 (HTTP) and 443(HTTPS)

HTTP History

FeatureHTTP/1.x HTTP/2 HTTP/3
Year establishedHTTP/1.0 came in 1996. HTTP/1.1, came in 1997.20152022
TCP connectionIn HTTP/1.0, each request needs seperate TCP connection. In HTTP/1.1, multiple requests can be sent in same TCP connection (keep alive).Multiple requests can be sent in one TCP connectionTCP is not used. QUIC is used
Chunked transfer encodingChunked transfer encoding is availableChunked transfer encoding is not available. Instead Bidirectional data streaming is used.Bidrection streaming is used
Binary headersHeader are sent in textual formatHeaders are compressed and sent in binary formatHeaders are compressed and sent in binary format
HOL blockingHOL existsHOL exists at TCP levelHOL completely removed
Server pushNAExistsNA

Please note that When the number of allowed parallel requests in the browser is used up, subsequent requests need to wait for the former ones to complete. That is called as Head of Line problem. only HEAD and some GET requests could be pipelined in a safe and idempotent mode. After many years of struggling with the problems introduced by enabling pipelining, this feature was first disabled and then removed from most browsers also because of the announced adoption of HTTP/2.

HTTP Sample example

HTTP Request and response have header and body.

POST /users HTTP/1.1
Content-Type: application/json
Authorization: Basic Og==
User-Agent: PostmanRuntime/7.29.0
Accept: */*
Postman-Token: 5b90742f-557d-4dc3-96a3-50aa1ef03254
Host: jsonplaceholder.typicode.com
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 28
 
{
"name" : "Donald trump"
}
 
HTTP/1.1 201 Created
Date: Fri, 22 Apr 2022 11:56:39 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 40
Connection: keep-alive
X-Powered-By: Express
X-Ratelimit-Limit: 1000
X-Ratelimit-Remaining: 999
X-Ratelimit-Reset: 1650628647
Vary: Origin, X-HTTP-Method-Override, Accept-Encoding
Access-Control-Allow-Credentials: true
Cache-Control: no-cache
Pragma: no-cache
Expires: -1
Access-Control-Expose-Headers: Location
Location: http://jsonplaceholder.typicode.com/users/11
X-Content-Type-Options: nosniff
Etag: W/"28-rgvCndyKGB08dJiIWr1C173Duts"
Via: 1.1 vegur
CF-Cache-Status: DYNAMIC
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/expect-ct"
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 6ffe2d677bd85a5b-MEL
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
 
{
"name": "Donald trump",
"id": 11
}


GET /users/1 HTTP/1.1
Content-Type: application/json
Authorization: Basic Og==
User-Agent: PostmanRuntime/7.29.0
Accept: */*
Postman-Token: f78b5876-e2ef-434a-801d-962336b3e87e
Host: jsonplaceholder.typicode.com
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
 
HTTP/1.1 200 OK
Date: Fri, 22 Apr 2022 11:58:16 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Powered-By: Express
X-Ratelimit-Limit: 1000
X-Ratelimit-Remaining: 999
X-Ratelimit-Reset: 1650628707
Vary: Origin, Accept-Encoding
Access-Control-Allow-Credentials: true
Cache-Control: no-cache
Pragma: no-cache
Expires: -1
X-Content-Type-Options: nosniff
Etag: W/"1fd-+2Y3G3w049iSZtw5t1mzSnunngE"
Via: 1.1 vegur
CF-Cache-Status: BYPASS
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/"
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 6ffe2fc35d255a5b-MEL
Content-Encoding: br
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
 
{
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret",
  "email": "[email protected]",
  "address": {
    "street": "Kulas Light",
    "suite": "Apt. 556",
    "city": "Gwenborough",
    "zipcode": "92998-3874",
    "geo": {
      "lat": "-37.3159",
      "lng": "81.1496"
    }
  },
  "phone": "1-770-736-8031 x56442",
  "website": "hildegard.org",
  "company": {
    "name": "Romaguera-Crona",
    "catchPhrase": "Multi-layered client-server neural-net",
    "bs": "harness real-time e-markets"
  }
}

HTTP Methods

  • GET
  • POST
  • PUT
  • PATCH
  • DELETE
  • CONNECT
  • TRACE
  • HEAD
  • OPTIONS
HTTP method has below properties
  • Request has payload body - Every method request can have payload except TRACE
  • Response has payload body - Every method response has payload except HEAD
  • Safe - GET, HEAD, OPTIONS, TRACE
  • Idempotent - GET, HEAD, PUT, DELETE, OPTIONS, TRACE
  • Cacheable - GET, HEAD, POST

Important concepts in HTTP

Caching

Caching is used to reduce the load on server and also to speed up the browsing experience of User. When client sends the request, browser will first check if the cached version of the resource is available in cache. If not, then only request is sent to server. Also it is checked if cached version is fresh or stale and not modified on the server. Various cache headers are used to manage cache.

Compression

Main purpose of compression is to improve transfer speed and bandwidth utilization. Server will send compressed data only if browser is able to decompress it. Most popular way to compress data is gzip, deflate, br, xz, bzip2. Browser can tell the server what encoding it supports using header "Accept-Encoding" Server will look at this header value and then take compression action accordingly. Just look at Content-Encoding response header to find out if the content is compressed.

HTTP vs HTTPS

Media Type (MIME Type)

Content-Type header tells what type of content is being transmitted in body. Some command types are given below.
  • application/json
  • application/javascript
  • application/pdf
  • application/x-www-form-urlencoded
  • application/xml
  • application/zip
  • audio/mpeg
  • video/mp4
  • image/jpeg, image/svg+xml, image/png
  • multipart/form-data
  • text/css
  • text/csv
  • text/html
  • text/xml - instead of this, use application/xml
  • text/plain
  • application/octet-stream - binary data
  • font/woff
  • application/vnd.ms-excel
  • application/x-7z-compressed

Encoding

Cookie

Byte Serving

CORS

Cross Site Scripting

HTTP Headers

In HTTP version 1.x, header fields are transmitted after the request line or the response line. Header fields are colon-separated key-value pairs in clear-text string format, terminated by a carriage return (CR) and line feed (LF) character sequence. The end of the header section is indicated by an empty line. HTTP/2 and HTTP/3 instead use a binary protocol, where headers are encoded in a single HEADERS and zero or more CONTINUATION frames using HPACK (HTTP/2) or QPACK (HTTP/3), which both provide efficient header compression. The request or response line from HTTP/1 has also been replaced by several pseudo-header fields, each beginning with a colon (:). Headers can be divided into 2 categories.
  • Request Headers
  • Response Headers

HTTP Request Headers

  • Accept: text/html
  • Accept-Encoding: gzip, deflate
  • Accept-Language: en-US
  • Accept-Charset: utf-8
  • Host: en.wikipedia.org:8080 - Mandatory since HTTP/1.1
  • Accept-Datetime: Thu, 31 May 2007 20:35:00 GMT
  • Date: Tue, 15 Nov 1994 08:12:31 GMT
  • User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:12.0) Gecko/20100101 Firefox/12.0
  • Content-Encoding: gzip
  • Content-Length: 348
  • Content-Type: Well known valiues are - application/json, text/html, application/xml, application/javascript, text/plain, application/octet-stream, multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW, application/x-www-form-urlencoded
  • Access-Control-Request-Method: GET
  • Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==
  • Cache-Control: no-cache
  • Connection: keep-alive - Do not use with HTTP/2
  • Cookie: $Version=1; Skin=new;
  • Forwarded: for=192.0.2.60;proto=http;by=203.0.113.43
  • From: [email protected]
  • HTTP2-Settings: token64
  • If-Match: "737060cd8c284d8af7ad3082f209582d"
  • If-Modified-Since: Sat, 29 Oct 1994 19:43:31 GMT
  • If-None-Match: "737060cd8c284d8af7ad3082f209582d"
  • If-Range: "737060cd8c284d8af7ad3082f209582d"
  • Range: bytes=500-999
  • Transfer-Encoding: chunked - Do not use with HTTP/2
  • If-Unmodified-Since: Sat, 29 Oct 1994 19:43:31 GMT
  • Max-Forwards: 10
  • Expect: 100-continue
  • Origin: http://www.xyz.com - Initiates the request for cross origin resource sharing
  • Pragma: no-cache
  • Prefer: return=representation
  • Proxy-Authorization: Basic djhwuejjsttpol==
  • Referer: http://en.wikipedia.org/wiki/Main_Page
  • TE: trailers, deflate
  • Trailer: Max-Forwards
  • Upgrade: h2c, HTTPS/1.3, IRC/6.9, RTA/x11, websockets
  • Via: 1.0 fred, 1.1 example.com (Apache/1.1)
  • Warning: 199 Miscellaneous warning
There can be other non standard HTTP Headers as well.
  • X-Forwarded-For: client1, proxy1, proxy2
  • X-Forwarded-Host: en.wikipedia.org:8080
  • X-Forwarded-Proto: https
  • X-Csrf-Token: i8XNjC4b8KVok4uw5RftR38Wgp2BFwql
  • X-Request-ID: f058ebd6-02f7-4d3f-942e-904344e8cde5

HTTP Response Headers

  • Connection: Keep-Alive - Use same TCP connection
  • Keep-Alive: timeout=5, max=1000 - Maximum 1000 requests can be made and idle timeout is 5
  • Set-Cookie: cookieName=cookieValue
  • Access-Control-Allow-Origin: *
  • Accept-Ranges: bytes
  • WWW-Authenticate: Basic realm="User Visible Realm" - This header is sent when client has not sent Authorization header with correct credentials.
  • Cache-Control: max-age=3600 - no-cache tells the browser and proxies to validate the cache content with the server before using it (this is done by using If-Modified-Since, If-Unmodified-Since, If-Match, If-None-Match attributes mentioned above). Sending a no-cache value thus instructs a browser or proxy to not use the cache contents merely based on "freshness criteria" of the cache content. Another common way to prevent old content from being shown to the user without validation is Cache-Control: max-age=0. This instructs the user agent that the content is stale and should be validated before use. The header field Cache-Control: no-store is intended to instruct a browser application to make a best effort not to write it to disk (i.e not to cache it)
  • Age: 12
  • Allow: GET, HEAD
  • Alt-Svc: http/1.1="http2.example.com:8001"; ma=7200
  • Location: http://www.w3.org/pub/WWW/People.html
  • Accept-Patch: text/example;charset=utf-8
  • Server: Apache/2.4.1 (Unix)
  • Connection: close - Do not use with HTTP/2
  • Content-Disposition: attachment; filename="fname.ext"
  • Content-Encoding: gzip
  • Content-Language: da
  • Content-Length: 348
  • Content-Location: /index.htm
  • Accept-CH: UA, Platform
  • Content-Range: bytes 21010-47021/47022
  • Content-Type: text/html; charset=utf-8
  • Date: Tue, 15 Nov 1994 08:12:31 GMT
  • Delta-Base: "abc"
  • ETag: "737060cd8c284d8af7ad3082f209582d"
  • Expires: Thu, 01 Dec 1994 16:00:00 GMT
  • IM: feed
  • Last-Modified: Tue, 15 Nov 1994 12:45:26 GMT
  • Link
  • Strict-Transport-Security: max-age=16070400; includeSubDomains
  • Tk: ?
  • Trailer: Max-Forwards
  • Transfer-Encoding: chunked
  • Pragma: no-cache
  • Preference-Applied: return=representation
  • Proxy-Authenticate: Basic
  • Public-Key-Pins: max-age=2592000; pin-sha256="E9CZ9INDbd+2eRQozYqqbQ2yXLVKB9+xcprMF+44U1g=";
  • Retry-After: 120
  • Upgrade: h2c, HTTPS/1.3, IRC/6.9, RTA/x11, websocket
  • Vary: *
  • Via: 1.0 fred, 1.1 example.com (Apache/1.1)
  • Warning: 199 Miscellaneous warning
There are several non standard headers as mentioned below.
  • Content-Security-Policy
  • X-WebKit-CSP: default-src 'self'
  • Permissions-Policy: fullscreen=(), camera=(), microphone=(), geolocation=(), interest-cohort=()
  • Refresh: 5; url=http://www.w3.org/pub/WWW/People.html
  • Status: 200 OK
  • Timing-Allow-Origin: *
  • X-Powered-By: PHP/5.4.0
  • X-XSS-Protection: 1; mode=block

Cookie

Cookies are created by server and stored at the client side i.e. the browser. Cookies are used for mainly 3 purposes.
  • Authentication - Session cookies are used to maintain stateful session between client and server
  • Advertising - tracking - These type of cookies are used by Ad companies like FB, Google.
  • Personal customized experience to web users - You can also use cookies to serve the content based on user's preferences.
Cookie can be of below types.
  • Same Site cookie - Cookie set by same website you are visiting
  • Third party cookie - Cookie set by different website than the one you are visiting
  • Session cookie - Temporary cookies that are destroyed when user logs out or closes the browser
  • Persistent cookie - Cookies that last until expiry date
  • Secure cookie - Cookies that can be served only on HTTPS
  • http-only cookie - Cookies that can not be accessed using JS. This is used to prevent XSS attacks.
  • Zombie cookie - Cookies that are stored in different location and recreated.
Cookie attributes
  • domain - Used to specify domains for which the cookies will be sent
  • path - used to specify for which paths the cookies will be sent
  • expires - indicates when cookie will be expired
  • max-age - indicates duration in seconds after which cookie will expire
  • secure
  • http-only
Cookie example - Server set cookie with name ID and value xyz

HTTP/1.0 200 OK
Set-Cookie: ID=xyz; Domain=.abc.com; Path=/; Expires=Wed, 09 Jan 2019 22:23:01 GMT; HttpOnly

Client sent back cookie when requesting the resource of same server.

GET /pqr.html HTTP/1.1
Host: www.abc.com
Cookie: ID=xyz

Cookie related attacks
  • DNS Cache poisoning
  • Cross Site scripting - XSS
  • Cross Site Request Forgery
Cookie laternatives
  • JWT tokens - Need to be sent to server explicitly
  • HTTP authentication - Basic authentication can be used to identify user. But will need login every time
  • Query string - Server sends the links with session identifers in query string
  • Hidden form fields
  • Web storage - local storage and session storage
  • ETag - used for Web cache validatio

HTTP Status Codes

  • 1XX (informational) - The request was received, continuing process.
  • 2XX (successful) - The request was successfully received, understood, and accepted.
  • 3XX (redirection) - Further action needs to be taken in order to complete the request.
  • 4XX (client error) - The request contains bad syntax or cannot be fulfilled.
  • 5XX (server error) - The server failed to fulfill an apparently valid request.

1XX - Informational Codes

  • 100 Continue - this error code means that server has received the valid headers and
  • client can now send the actual payload
  • 101 Switching Protocols - Server is ready to change protocols
  • 102 Processing - used to indicate that server is still processing the request
  • 103 Early Hints - sends response headers before final HTTP message

2XX - Success Codes

  • 200 OK - All good
  • 201 Created - New resource created on server
  • 202 Accepted - Request is processing or may not be completed
  • 203 Non-Authoritative Information - modified response is being sent
  • 204 No Content - Request processed but no content in the body
  • 205 Reset Content - Requester reset its document view, and is not returning any content.[14]
  • 206 Partial Content - Partial content of the resource is being shared
  • 207 Multi-Status - returns xml body with multiple status codes
  • 208 Already Reported
  • 226 IM Used - Indicates if instance manipulation is used

3XX - Redirection Error Codes

  • 300 - multiple choices - resource is available in multiple formats
  • 301 - resource moved permanently
  • 304 - Not Modified
  • 305 - Use Proxy

4XX - Client Errors

  • 400 - Bad Request
  • 401 - unauthorized - No authentication provided by client
  • 402 - Payment required
  • 403 - Forbidden - No permission to access resource
  • 404 - not found
  • 405 - Method not allowed
  • 406 - Not Acceptable - Server is not able to serve the content requested by Client
  • 407 - proxy authentication required
  • 429 - Too many requests
  • 416 - Range not satisfiable
  • Unsupported Media Type

5XX - Server Errors

  • 500 - Internal Server Error
  • 501 - Not implemented
  • 502 - Bad Gateway
  • 503 - Service Unavailable
  • 504 - Gateway Timeout

HTTP GET vs POST

POST method can be used to
  • Send form data in key value pair
  • Send query string data in key value pair
  • Sensitive data like password must be sent in the form data.
  • ASCII as well as binary data or files can be sent

Only drawback of POST is that Submissions by form with POST cannot be bookmarked.

GET method has several restricitions.
  • wipes query parameters and if form data is available, that is converted into query string parameters
  • The length of the URL is limited by 2048 characters
  • Bookmarking of url is possible and should be used with non secure data
  • Binary data can not be sent

HTTP Body

HTTP Interview Questions and Answers

  • How to know if web server is using which version of HTTP protocol - Go to response headers in chrome dev tools and click on view source.
  • How to configure node server to use specific version of HTTP? - Use NGINX
  • How to view raw http request and response - use postman console or chrome dev tools

Web development and Automation testing

solutions delivered!!