Skip to main content

New Lines - Windows/Unix/Linux/MacOS - viM

If you deal with scripts and other text files and move between platforms you probably discovered this "issue".
Only the founding developers can explain why they chose what they did - googling about will show you a couple of different explanations - whatever the reasons, here are the differences and how to convert.

The formats

The Characters in use (referenced in OS info below)

  • LF
  • zoeyburger
    • Usually referred to as LF  or Line feed
    • Ascii code decimal 10
    • Hex: A or 0xA
    • Octal: 12 or O12
    • Typical Escaped character in many shells and languages: \n
  • CR
    • Usually referred to as CR or Carriage Return
    • Ascii code decimal 13
    • Hex D or 0xD
    • Typical Escaped character in many shells and languages: \r

Unix, Linux, and Modern MacOS - The POSIX standard

Each Line ends with a single character:  LF
Most programming languages will understand/interpret this format properly.
Simple Windows programs, like the built in Notepad will not show this properly.

Windows (and DOS)

Each line ends with two consecutive characters in this order: CRLF

Classic (Ancient) MacOS

Each Line ends with a single character:  CR
This is no longer much of a practical issue, it is from the dark ages of Mac.

Windows - Use Advanced (Free) Editors

If you primarily work on Windows, then all you need is a good editor of sorts, like Notepad++ or Visual Studio Code - they can handle conversion and detection for you. If you are in a hurry and just annoyed that a file opens in Notepad and looks like one long line, just open the file in Wordpad instead, that shoul let you read it. (But really, install Notepad++)

Vim (vi Improved)

vi is the "native" CLI/Shell based editor for many Unix and Linux users, it has been around for a long time, vim is the modern and quite capable version - and it is my go-to on any non-windows system, any CLI for sure, now even exists for Windows with a graphical UI (gvim).
It is not an easy editor to learn in a hurry, you need to take some time reading, testing, and understanding how it is fundamentally different (than say pico or nano), vi is not one to figure out by clicking around. That said, once you grasp what it does, it has great powers that will to the untrained eye seem magical.

If you open a file on a system and you see a bunch of ^M at each line ending, here is how you can replace them all in vi with nothing, so that you end up with LF only endings:

  • start a command with colon :
  • % for all lines
  • s/ for substitute
  • Ctrl+v, then hit Enter - this will produce a ^M
  • then // to substitute with nothing
  • so your command will look like:  % s/^M//

Shell / CLI

There are a ton of ways to do these conversions - here are a couple options

  • To create LF only from CRLF (Simply delete any and all CR)
    • cat windowsfile | sed 's/\r//' > nixfile
    • cat windowsfile | tr -d '\015' > nixfile
    • cat windowsfile | dos2unix > nixfile

HTTP headers and MIME Formatting

This has nothing to do with files and scripts editors - but just though i would point out that the standards used in MIME (email) formatting and HTTP headers utilizes CRLR (\r\n), it is a common mistake for unseasoned script writers and such to forget this..  Make sure to read up on the rules and RFCs if you are creating mail senders or http services.

Comments

Popular posts from this blog

Introducing Sau MGM - Small to medium Business Information technology management

I am (slowly) working on a project called SauMGM - a small/medium business IT-Department administration utility, database and more.
I will also use this blog to post helpful hints and tricks, some logs of things I have been doing, as well as a place to just store things for myself, such as remembering how to do certain things. I ofetn find myself not remembering the exact syntax on things i do occasionally, like openssl specifics.
I have been doing systems and network administration since 1999, and I am still very much hands on in all kinds of projects and technologies.

Home made SAN Migration

The topic sounds more elaborate than it is - alternative title could be "Hackjob SAN volume backup and restore".
The SetupIn a legacy-style SAN and Compute setup, I have an EMC Unity 450F box deployed with Fibre Channel (FC) to a Cisco UCS (Unified Compute System). I am booting the UCS blades off the SAN, running vmware with Block/LUN DataStores, and one blade running Windows.
I also have a Dell R740 server in the mix, with a Qlogic HBA as well as onboard storage.
The Situation Not in production yet, but we had spun up a few VMs, and all our blades had been installed, esx and vcenter running, a few VM's, and including the physical Windows blade, and the R740.

Then we discovered that the Unity had no SFP+ ports in it, and I need to do replication - swearing my vendor up and down, I call EMC, and they are sending out NIC modules and guy to install it. BUT, because we have to remove a module to insert a new one, the whole SAN box has to be reset to Factory setting (!!!). My…