cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Anonymous
by Not applicable
  • 1753 Views
  • 1 replies
  • 0 kudos

E2 Workspace DNS Unreachable

My E2 workspace is in a RUNNING state, but the DNS is unreachable.

  • 1753 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Databricks Employee
  • 0 kudos

Try deleting the RUNNING workspace, wait for 5-10 minutes, and recreate the same workspace. If that doesn't solve the problem, file a support ticket

  • 0 kudos
Anonymous
by Not applicable
  • 1644 Views
  • 1 replies
  • 0 kudos

E2 workspace - Error Message Malformed Request : Invalid xxx in the HTTP request body

I received one of the following errors: MALFORMED_REQUEST: Invalid xxx in the HTTP request body or MALFORMED_REQUEST: Invalid xxx in body, where xxx is credentials, storage configurations, networks, etc.

  • 1644 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Databricks Employee
  • 0 kudos

It denotes that the input payload is not what is expected the api-endpoint for the e2 accounts api. Possible causes include typo in variable values or json formatting issues ( not providing quotes etc )

  • 0 kudos
Anonymous
by Not applicable
  • 7265 Views
  • 1 replies
  • 1 kudos
  • 7265 Views
  • 1 replies
  • 1 kudos
Latest Reply
sajith_appukutt
Databricks Employee
  • 1 kudos

To access these driver log files from the UI, you could go to the Driver Logs tab on the cluster details page. You can also configure a log delivery location for the cluster. Both worker and cluster logs are delivered to the location you specify.

  • 1 kudos
Anonymous
by Not applicable
  • 1946 Views
  • 1 replies
  • 0 kudos
  • 1946 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Databricks Employee
  • 0 kudos

When you remove a user from Databricks, a special backup folder is created in the workspace. More details at https://kb.databricks.com/notebooks/get-notebooks-deleted-user.html

  • 0 kudos
Anonymous
by Not applicable
  • 1405 Views
  • 1 replies
  • 0 kudos
  • 1405 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Databricks Employee
  • 0 kudos

If you have some of the features like "secure cluster connectivity", "multi-workspace accounts" , chances are that the account is E2. Would recommend to check with your Databricks accounts team.

  • 0 kudos
Anonymous
by Not applicable
  • 1735 Views
  • 1 replies
  • 0 kudos

Saving charts in S3

Can you save the charts that you create in Databricks notebooks as an image file to s3? For example I'm viewing the dataframe using display(df) and then using the plot button to actually create the graph

  • 1735 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Databricks Employee
  • 0 kudos

Would rendering visualizations via plotly and saving them to s3 work ?

  • 0 kudos
Anonymous
by Not applicable
  • 2840 Views
  • 1 replies
  • 1 kudos

SQL Formatting

Is there a way for individual users to adjust the SQL formatting defaults?

  • 2840 Views
  • 1 replies
  • 1 kudos
Latest Reply
User16826994223
Databricks Employee
  • 1 kudos

Do you want a different format than Databrick provide the formatting in Sql cell?

  • 1 kudos
User16765131552
by Databricks Employee
  • 2076 Views
  • 2 replies
  • 0 kudos

Resolved! Does Azure Databricks and Delta Layer make it a Lakehouse?

Even after going through many resources, I have failed to understand what constitutes a lakehouse, hence my question below.If we have Azure Gen 2 Storage, ADF, and Azure Databricks with the possibility of converting the incoming CSV files into Delta ...

  • 2076 Views
  • 2 replies
  • 0 kudos
Latest Reply
User16826994223
Databricks Employee
  • 0 kudos

Lakehouse is a concept defined with the following Parameter-Data is stored in an open standard format.Data is stored in a way which support Data Science,ML and BI loads.Delta is just a way or engine on cloud storage that provides control on data and...

  • 0 kudos
1 More Replies
Anonymous
by Not applicable
  • 1950 Views
  • 2 replies
  • 1 kudos

What Databricks Runtime will I have to use if I want to leverage Python 2?

I have some code which is dependent on python 2. I am not able to use Python 2 with Databricks runtime 6.0.

  • 1950 Views
  • 2 replies
  • 1 kudos
Latest Reply
User16826994223
Databricks Employee
  • 1 kudos

When you create a Databricks Runtime 5.5 LTS cluster by using the workspace UI, the default is Python 3. You have the option to specify Python 2. If you use the Databricks REST API to create a cluster using Databricks Runtime 5.5 LTS, the default is ...

  • 1 kudos
1 More Replies
User16826994223
by Databricks Employee
  • 1650 Views
  • 1 replies
  • 0 kudos

How is the ETL process different than trigger once stream

I am little confused between what to use between structured stream(trigger once) and etl batch jobs, can I get help here on which basis i should make my decision.

  • 1650 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Databricks Employee
  • 0 kudos

In Structured Streaming, triggers are used to specify how often a streaming query should produce results. A RunOnce trigger will fire only once and then will stop the query - effectively running it like a batch job.Now, If your source data is a strea...

  • 0 kudos
User15787040559
by Databricks Employee
  • 4668 Views
  • 1 replies
  • 0 kudos

What's the difference between Normalization and Standardization?

Normalization typically means rescales the values into a range of [0,1].Standardization typically means rescales data to have a mean of 0 and a standard deviation of 1 (unit variance).

  • 4668 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16826994223
Databricks Employee
  • 0 kudos

Normalization typically means rescales the values into a range of [0,1]. Standardization typically means rescales data to have a mean of 0 and a standard deviation of 1 (unit variance).A link which explains better is - https://towardsdatascience.com...

  • 0 kudos
User16826994223
by Databricks Employee
  • 1105 Views
  • 1 replies
  • 2 kudos

Issue: Your account {email} does not have the owner or contributor role on the Databricks workspace resource in the Azure portal 

Issue: Your account {email} does not have the owner or contributor role on the Databricks workspace resource in the Azure portal

  • 1105 Views
  • 1 replies
  • 2 kudos
Latest Reply
sajith_appukutt
Databricks Employee
  • 2 kudos

https://docs.microsoft.com/en-us/azure/databricks/scenarios/frequently-asked-questions-databricks#solution-1

  • 2 kudos
User16826994223
by Databricks Employee
  • 2468 Views
  • 1 replies
  • 0 kudos

Streaming with Kafka with the same groupid

A kafka topic is having 300 partitions and I see two clusters are running and have the same group id, will the data be duplicate in my delta bonze layer

  • 2468 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Databricks Employee
  • 0 kudos

By default, each streaming query generates a unique group ID for reading data ( ensuring it's own  its own consumer group ) . In scenarios where you'd want to specify it (authz etc ) , it is not recommended to have two streaming applications specify ...

  • 0 kudos
User16826994223
by Databricks Employee
  • 6329 Views
  • 3 replies
  • 0 kudos

Resolved! Delta lake Check points storage concept

In which format the Checkpoints are stored in storage and , how does it help in delta to increase performance.

  • 6329 Views
  • 3 replies
  • 0 kudos
Latest Reply
aladda
Databricks Employee
  • 0 kudos

Great points above on how checkpointing helps with performance. In additional Delta Lake also provides other data organization strategies such as compaction, Z-ordering to help with both read and write performance of Delta Tables. Additional details ...

  • 0 kudos
2 More Replies
Srikanth_Gupta_
by Databricks Employee
  • 3811 Views
  • 2 replies
  • 0 kudos
  • 3811 Views
  • 2 replies
  • 0 kudos
Latest Reply
aladda
Databricks Employee
  • 0 kudos

Temp Views and Global Temp Views are the most common way of sharing data across languages within a Notebook/Cluster

  • 0 kudos
1 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels