- 1030 Views
- 1 replies
- 0 kudos
Koalas or Pyspark
Should I use PySpark’s DataFrame API or Koalas, Which one is recommended , is there any performance impact if i use koalas or little slower than pyspark API
- 1030 Views
- 1 replies
- 0 kudos
- 0 kudos
If you are already familiar with pandas and want to leverage Spark for big data, we recommend using Koalas. If you are learning Spark from ground up, we recommend you start with PySpark’s API.
- 0 kudos
- 1246 Views
- 1 replies
- 0 kudos
Photon usage
How do I know how much of a query/job used Photon?
- 1246 Views
- 1 replies
- 0 kudos
- 0 kudos
If you are using Photon on Databricks SQLClick the Query History icon on the sidebar.Click the line containing the query you’d like to analyze.On the Query Details pop-up, click Execution Details.Look at the Task Time in Photon metric at the bottom.
- 0 kudos
- 3328 Views
- 1 replies
- 0 kudos
Malformed Request Error Message
I received the following error when launching a workspace: MALFORMED_REQUEST: Failed storage configuration validation checks: PUT, LIST, DELETE. How do I fix this?
- 3328 Views
- 1 replies
- 0 kudos
- 0 kudos
Check the S3 bucket policy and region. It looks like storage config validation is failing
- 0 kudos
- 1620 Views
- 1 replies
- 0 kudos
Failed E2 Workspace Error Message
My workspace ended up in a FAILED workspace state with one of the following messages:INVALID_STATE: The maximum number of VPCs has been reached.INVALID_STATE: The maximum number of VPC endpoints has been reached.INVALID_STATE: The maximum number of a...
- 1620 Views
- 1 replies
- 0 kudos
- 0 kudos
The maximum number of xxx has been reached indicates that you have hit the soft limits for some of the AWS resources in that region. These are mostly soft limits and you could file a request to AWS support team to increase this
- 0 kudos
- 1756 Views
- 1 replies
- 0 kudos
E2 Workspace DNS Unreachable
My E2 workspace is in a RUNNING state, but the DNS is unreachable.
- 1756 Views
- 1 replies
- 0 kudos
- 0 kudos
Try deleting the RUNNING workspace, wait for 5-10 minutes, and recreate the same workspace. If that doesn't solve the problem, file a support ticket
- 0 kudos
- 1644 Views
- 1 replies
- 0 kudos
E2 workspace - Error Message Malformed Request : Invalid xxx in the HTTP request body
I received one of the following errors: MALFORMED_REQUEST: Invalid xxx in the HTTP request body or MALFORMED_REQUEST: Invalid xxx in body, where xxx is credentials, storage configurations, networks, etc.
- 1644 Views
- 1 replies
- 0 kudos
- 0 kudos
It denotes that the input payload is not what is expected the api-endpoint for the e2 accounts api. Possible causes include typo in variable values or json formatting issues ( not providing quotes etc )
- 0 kudos
- 7269 Views
- 1 replies
- 1 kudos
- 7269 Views
- 1 replies
- 1 kudos
- 1 kudos
To access these driver log files from the UI, you could go to the Driver Logs tab on the cluster details page. You can also configure a log delivery location for the cluster. Both worker and cluster logs are delivered to the location you specify.
- 1 kudos
- 1949 Views
- 1 replies
- 0 kudos
- 1949 Views
- 1 replies
- 0 kudos
- 0 kudos
When you remove a user from Databricks, a special backup folder is created in the workspace. More details at https://kb.databricks.com/notebooks/get-notebooks-deleted-user.html
- 0 kudos
- 1405 Views
- 1 replies
- 0 kudos
- 1405 Views
- 1 replies
- 0 kudos
- 0 kudos
If you have some of the features like "secure cluster connectivity", "multi-workspace accounts" , chances are that the account is E2. Would recommend to check with your Databricks accounts team.
- 0 kudos
- 1736 Views
- 1 replies
- 0 kudos
Saving charts in S3
Can you save the charts that you create in Databricks notebooks as an image file to s3? For example I'm viewing the dataframe using display(df) and then using the plot button to actually create the graph
- 1736 Views
- 1 replies
- 0 kudos
- 0 kudos
Would rendering visualizations via plotly and saving them to s3 work ?
- 0 kudos
- 2841 Views
- 1 replies
- 1 kudos
SQL Formatting
Is there a way for individual users to adjust the SQL formatting defaults?
- 2841 Views
- 1 replies
- 1 kudos
- 1 kudos
Do you want a different format than Databrick provide the formatting in Sql cell?
- 1 kudos
- 2077 Views
- 2 replies
- 0 kudos
Resolved! Does Azure Databricks and Delta Layer make it a Lakehouse?
Even after going through many resources, I have failed to understand what constitutes a lakehouse, hence my question below.If we have Azure Gen 2 Storage, ADF, and Azure Databricks with the possibility of converting the incoming CSV files into Delta ...
- 2077 Views
- 2 replies
- 0 kudos
- 0 kudos
Lakehouse is a concept defined with the following Parameter-Data is stored in an open standard format.Data is stored in a way which support Data Science,ML and BI loads.Delta is just a way or engine on cloud storage that provides control on data and...
- 0 kudos
- 1950 Views
- 2 replies
- 1 kudos
What Databricks Runtime will I have to use if I want to leverage Python 2?
I have some code which is dependent on python 2. I am not able to use Python 2 with Databricks runtime 6.0.
- 1950 Views
- 2 replies
- 1 kudos
- 1 kudos
When you create a Databricks Runtime 5.5 LTS cluster by using the workspace UI, the default is Python 3. You have the option to specify Python 2. If you use the Databricks REST API to create a cluster using Databricks Runtime 5.5 LTS, the default is ...
- 1 kudos
- 1651 Views
- 1 replies
- 0 kudos
How is the ETL process different than trigger once stream
I am little confused between what to use between structured stream(trigger once) and etl batch jobs, can I get help here on which basis i should make my decision.
- 1651 Views
- 1 replies
- 0 kudos
- 0 kudos
In Structured Streaming, triggers are used to specify how often a streaming query should produce results. A RunOnce trigger will fire only once and then will stop the query - effectively running it like a batch job.Now, If your source data is a strea...
- 0 kudos
- 4672 Views
- 1 replies
- 0 kudos
What's the difference between Normalization and Standardization?
Normalization typically means rescales the values into a range of [0,1].Standardization typically means rescales data to have a mean of 0 and a standard deviation of 1 (unit variance).
- 4672 Views
- 1 replies
- 0 kudos
- 0 kudos
Normalization typically means rescales the values into a range of [0,1]. Standardization typically means rescales data to have a mean of 0 and a standard deviation of 1 (unit variance).A link which explains better is - https://towardsdatascience.com...
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 485 | |
| 349 | |
| 287 |
