Themyscira Wireless Technical Specification TW-TS-005 Version 1.1.0 Hexadecimal file format for sequences of RTP payloads for GSM speech codecs 0. Foreword This Technical Specification (TS) has been produced under the authority of the Presiding Sisterhood (government) of the Women's Republic of Themyscira as part of Themyscira Wireless technical initiative. Author: Mother Mychaela N. Falconia, High Priestess of Telecommunications As an official publication of Themyscira government, this document is not subject to copyright. 1. Scope Osmocom and Themyscira software components for GSM speech services represent the frames of each supported codec type as RTP payloads, using payload formats defined in various standards. In development and test activities related to these speech codecs and software components, a need sometimes arises to store sequences of RTP-encoded codec frames in a file at rest, perhaps representing a capture of a session or a test sequence. The present TS defines an ASCII hexadecimal file format that can represent RTP payloads that correspond to several different codec types used in GSM networks. 2. References The following documents contain provisions which, through reference in this text, constitute provisions of the present document. [1] IETF RFC 3550, H. Schulzrinne, S. Casner, R. Frederick, V. Jacobson: "RTP: A Transport Protocol for Real-Time Applications". [2] IETF RFC 3551, H. Schulzrinne, S. Casner: "RTP Profile for Audio and Video Conferences with Minimal Control". [3] ETSI TS 101 318: "Telecommunications and Internet Protocol Harmonization Over Networks (TIPHON); Using GSM speech codecs within ITU-T Recommendation H.323". [4] TW-TS-001: "Enhanced RTP transport of FR and EFR codec frames in an IP-based GSM RAN". [5] TW-TS-002: "Enhanced RTP transport of HRv1 codec frames in an IP-based GSM RAN". [6] IETF RFC 5993, X. Duan, S. Wang, M. Westerlund, K. Hellwig, T. Johansson: "RTP Payload Format for Global System for Mobile Communications Half Rate (GSM-HR)". [7] 3GPP TS 48.103: "Base Station System - Media GateWay (BSS-MGW) interface; User Plane transport mechanism". [8] 3GPP TS 26.102: "Adaptive Multi-Rate (AMR) speech codec; Interface to Iu, Uu and Nb". [9] IETF RFC 4867, J. Sjoberg, M. Westerlund, A. Lakaniemi, Q. Xie: "RTP Payload Format and File Storage Format for the Adaptive Multi-Rate (AMR) and Adaptive Multi-Rate Wideband (AMR-WB) Audio Codecs". [10] TW-TS-006: "Enhanced RTP transport of AMR-NB codec frames in an IP-based GSM RAN". [11] TW-TS-007: "Compressed format for RTP transport of GSM CSData". 3. Supported codec types and RTP formats The hexadecimal file format of the present TS was originally invented for the purpose of representing the following combinations of GSM speech codecs and their associated RTP formats: * FR, HR and EFR codecs in RTP encodings of ETSI TS 101 318 sections 5.1, 5.2 and 5.3, respectively; * FR and EFR codecs in the extended RTP format of TW-TS-001; * HR codec in 3GPP-5993 format, i.e., the payload format of RFC 5993 as constrained by 3GPP TS 26.102 referenced from TS 48.103; * HR codec in super-5993 format of TW-TS-002. Since that original definition, new applications of this hexadecimal file format have been found for AMR and for CSD, as detailed in Annexes C and D. Further applications may be found in the future, but none are envisioned currently. 4. File format definition 4.1. Principal definition The file format of the present TS is a line-based ASCII text format. Each line consists solely of printable ASCII characters, followed by OS-specific newline encoding. The maximum allowed length of each line is 80 characters, not including the OS-specific newline encoding. 4.2. Line record structure Each non-blank, non-comment line in an ASCII text file formatted per the present TS constitutes one record and represents a single RTP payload of a supported codec type. The RTP payload shall be written as a long string of hexadecimal digits, with two hex digits for each octet and no white space between octets. 4.2.1. Case of hexadecimal digits Hexadecimal digits A through F may be either uppercase or lowercase. 4.3. Frame delineation In all supported codec-format combinations listed in section 3, each 20 ms speech frame, be it good or errored, is transferred as a discrete RTP payload. In the hexadecimal file format of the present TS, each such frame or payload shall be stored as a separate hex-encoded line, terminated with OS-specific newline representation. Therefore, any program reading these hexadecimal files will have the same ability to detect and handle frame boundaries as a network element that receives a new RTP packet every 20 ms. 4.3.1. Representation of frame gaps Enhanced RTP transport formats of TW-TS-001 and TW-TS-002 require that a properly encoded frame be transmitted in every 20 ms window, and provide special No_Data frames to be used when the sender has nothing else to transmit. However, the same provision does not exist in other RTP formats, and the most common industry practice is to send nothing at all, or perhaps a zero-length payload, during frame gaps. In the hexadecimal file format of the present TS, a frame gap corresponding to an "empty" 20 ms time window for which no payload bytes exist is the keyword "NULL" (without quotes) in the place of a hexadecimal record. For consistency with case-insenstive hexadecimal digits (see section 4.2.1), this keyword shall likewise be recognized in a case-insensitive manner. 4.4. White space and comments Blank lines, whitespace-only lines or comment-only lines are allowed anywhere, and shall be ignored by programs that read frame sequence files per the present TS. White space is defined as any sequence of one or more ASCII space and/or tab characters. Comments are introduced with '#' character; all text following this character till the end of the line is to be treated as a comment and ignored. Each valid record (hexadecimal string or NULL keyword) shall start at the beginning of its respective physical line in the text file (i.e., no preceding white space is allowed), but may be followed by trailing white space. A record may also be followed by a comment (introduced with '#' character) - however, such record-attached comments must be preceded (separated from the record body) by white space. '#' directly after a record, without intervening white space, is not allowed. 5. Postel principle Programs that read frame sequence files per the present TS may implement parsers that are more permissive than the rules stated in chapter 4, accepting more than what is allowed by the strict language of the present TS. However, all programs that generate output that aims to follow the present TS shall strictly obey all file format rules enumerated in chapter 4. Annex A (normative): Representation format for FR and EFR codecs The term "TW-TS-005 Annex A file format" shall be used for data files that store a sequence of GSM-FR or GSM-EFR codec frames using the hexadecimal file format of the present TS. The following RTP formats are allowed for this Annex: * The basic RTP format of ETSI TS 101 318, also duplicated in RFC 3551; * The extended RTP format of TW-TS-001, either TEH only or a payload frame preceded by TEH; * A frame gap represented by NULL keyword. A program that reads FR/EFR frame sequence files per this Annex can autodetect the specific format in use out of the above, as well as detect invalid input, by checking the length of each record (stored RTP payload) and the signature nibbles in the leading octets: * An RTP payload of 1 octet can only be TW-TS-001 TEH in this context, in which case the upper nibble can only equal 0xE; * An RTP payload of 31 octets can only be EFR in basic RTP format, in which case the upper nibble of the first octet can only equal 0xC; * An RTP payload of 32 octets can only be EFR in extended RTP format: - the upper nibble of the first octet equals 0xE; - the upper nibble of the second octet equals 0xC; * An RTP payload of 33 octets can only be FR in basic RTP format, in which case the upper nibble of the first octet can only equal 0xD; * An RTP payload of 34 octets can only be FR in extended RTP format: - the upper nibble of the first octet equals 0xE; - the upper nibble of the second octet equals 0xD; * Any other input that does not equal any of the possibilities enumerated above is invalid. A 1-octet RTP payload that consists of TW-TS-001 TEH with NDF=1 is equally valid for FR and EFR and thus cannot be used to distinguish between the two codecs; all other valid possibilities are specific to either FR only or EFR only. Annex B (normative): Representation format for HR codec The term "TW-TS-005 Annex B file format" shall be used for data files that store a sequence of GSM-HR codec frames using the hexadecimal file format of the present TS. The following RTP formats are allowed for this Annex: * The basic RTP format of ETSI TS 101 318 section 5.2; * The extended RTP format of RFC 5993 and TW-TS-002 - as detailed in [5], many (but not all) super-5993 RTP payloads are also valid RFC 5993 payloads; * A frame gap represented by NULL keyword. A program that reads GSM-HR frame sequence files per this Annex can autodetect the specific format in use out of the above, as well as detect invalid input, by checking the length of each record (stored RTP payload) and for some lengths, certain bit fields in the first octet: * An RTP payload of 1 octet can only be TW-TS-002 FT=1 or FT=7 in this context; the single octet should be checked to verify that it is valid TW-TS-002 format with one of these two frame types. 1-octet RTP payloads with FT=1 should be checked to verify that ISV bit equals 0. * An RTP payload of 14 octets can only be TS 101 318 section 5.2 in this context. Because every possible pattern of 112 bits is a potentially valid GSM-HR codec frame, no further validation checks can be done on payloads of this length. * An RTP payload of 15 octets can only be RFC 5993 or TW-TS-002 in this context, with FT=0, FT=2 or FT=6. FT=1 and ISV=1 is also a valid possibility per updated TW-TS-002 spec. * Any other input that does not equal any of the possibilities enumerated above is invalid. Annex C (normative): Representation of RTP payloads for AMR C.1. RTP payload format options for GSM-AMR There exist 3 variants of RTP payload format for AMR-NB speech codec in IP-based GSM networks: * Format stipulated by 3GPP: RFC 4867 in bandwidth-efficient (BWE) mode, restricted to a single codec frame per RTP packet, as specified in TS 26.102; * Variant used by Osmocom: RFC 4867 in octet-aligned (OA) mode, as opposed to BWE, but with all other restrictions unchanged from TS 26.102; * Enhanced RTP format defined in TW-TS-006 [10], also known as OAX (OA extended) format. C.2. Application of the present TS to AMR A stream of GSM-AMR RTP payloads in any of the 3 possible formats enumerated in section C.1 may be recorded in a hexadecimal file per the present TS, subject to the following limitations: * OA format of RFC 4867 and OAX format of TW-TS-006 can be freely mixed, such that a process instructed to read TW-TS-005 Annex C OA/OAX hex files can reliably interpret from each payload whether or not it uses any of the extensions of TW-TS-006. However, no reliable automatic detection is possible between OA/OAX and BWE formats for AMR - therefore, out-of-band knowledge is required for each AMR hex file, indicating whether it captures OA/OAX or BWE payloads. * All TW-TS-006 OAX payloads that can be produced by conversion from TRAU frames of TS 48.060 or 48.061 fit into the 40 octet limit imposed by section 4.1 of the present TS. However, an IP-native network element that implements TW-TS-006 (an IP-native BTS or a TFO-capable speech transcoder) can emit a speech frame in the highest 12k2 mode with TFO config parameters included - a feat which is possible in TW-TS-006 RTP format but not in TRAU frames - and in that case the RTP payload will exceed the 40 octet limit. Therefore, the ability to capture GSM-AMR RTP payloads in the hexadecimal format of the present TS exists only when this unsupported combination is precluded. C.3. Relation to the storage format of RFC 4867 RFC 4867 specifies not only its complement of RTP payload formats, but also a storage format that is commonly known as .amr file format. The storage format of RFC 4867 is preferable to the hexadecimal format of the present TS and Annex when the objective is to store or manipulate speech or other audio recordings: the format is a widely accepted standard, it omits information that is superfluous to audio recording applications (CMR etc), and the binary format is more efficient. However, the hexadecimal format of the present TS and Annex is intended for a different class of applications: * During network debugging, it may be necessary to extract the sequence of payloads from a live or captured RTP stream. The present TS has been used very effectively to accomplish this feat with classic GSM speech codecs, and the present Annex extends the same ability to AMR. * In the context of unit tests for library functions that operate on RTP payload formats for AMR, the present TS and Annex offer the following advantages over RFC 4867 storage format: - Binary formats are often contraindicated for unit tests that need to be committed into a source code repository, whereas a line-based hexadecimal format is generally suitable. - For unit tests it is generally necessary to preserve the complete RTP payload in the test data file, including CMR, the BWE vs OA format distinction, and any possible extensions of TW-TS-006 - none of which are possible in the standard storage format of RFC 4867. It is advisable to develop utilities that convert between the hexadecimal format of the present TS and Annex and the standard (binary) storage format of RFC 4867. Conversion from TW-TS-005 Annex C hex format to RFC 4867 storage format would allow captured sequences of RTP payloads to be played as audio; conversion in the opposite direction can assist in preparation of unit test data. Anned D (normative): Representation of compressed-CSD RTP payloads Compressed-CSD RTP payloads defined in TW-TS-007 (3 different compressed RTP formats, distinguished by payload length) may be recorded in a hexadecimal file per the present TS, and a hex file of this form can serve as a capture of a CSData-carrying RTP session if the compressed format of TW-TS-007 was used. Unlike the compressed format of TW-TS-007, standard CLEARMODE payloads for CSD (a 160 byte payload every 20 ms) cannot be represented in the hexadecimal format of the present TS: this limitation exists because section 4.1 limits each record to 80 characters or 80 hexadecimal digits or 40 octets of RTP payload. However, a CSD session that took place using CLEARMODE format can be recorded in the compressed format of this Annex by applying the compression function of TW-TS-007 section 8.1 to each RTP payload to be saved, and a compressed hexadecimal recording per this Annex can be played into a CLEARMODE RTP stream by applying the decompression function of TW-TS-007 section 8.2 to each compressed payload. Annex E (informative): Specification change history Version 1.0.2: initial publication for review and early implementation by Themyscira and Osmocom communities. Version 1.0.3: correct a typing error in Annex A. Version 1.0.4: update Annex B for TW-TS-002 version 1.2.0. Version 1.1.0: add support for AMR and CSD; Annexes C and D are new.