- 7057 Views
- 1 replies
- 0 kudos
How to conver Dataframe into JSON on Databricks?
Can I convert my jdbc Dataframe into JSON ? Because when I tried it, it got an error. I'm using a script as Pandas DataFrame function df.to_json()
- 7057 Views
- 1 replies
- 0 kudos
- 0 kudos
df.toJSON()
- 0 kudos
- 7905 Views
- 3 replies
- 1 kudos
- 7905 Views
- 3 replies
- 1 kudos
- 1 kudos
The answer varies depending on the cloud provider (as of June 2021) . In GCP, since the architecture is based on GKE , there are additional ip requirements. For more details see
- 1 kudos
- 1497 Views
- 0 replies
- 0 kudos
Escaped quotes mess up table records
When table content is dumped from the RDBMS (e.g. Oracle), some column values may contain escaped double quotes (\") in the column values, which may cause the values from multiple columns to be concatenated into one value and result in corrupted reco...
- 1497 Views
- 0 replies
- 0 kudos
- 53047 Views
- 6 replies
- 0 kudos
Resolved! Job fails with "The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached."
No other output is available, not even output from cells that did run successfully. Also, I'm unable to connect to spark ui or view the logs. It makes an attempt to load each of them, but after some time an error message appears saying it's unable ...
- 53047 Views
- 6 replies
- 0 kudos
- 0 kudos
most of the time it's out of memory on driver node. check over all the drive log, data node log in Spark UI. And check if u r collecting huge data to drive node, e.g. collect()
- 0 kudos
- 1645 Views
- 1 replies
- 0 kudos
Delta - open source?
Delta is open source but certain features such as OPTIMIZE, ZORDER are only available on managed DBR. So how open sourced is it really?
- 1645 Views
- 1 replies
- 0 kudos
- 0 kudos
Some of the feature is exclusively added by datbricks on top of delta not by comunity so comapny has right whether it wants to open source or not
- 0 kudos
- 7472 Views
- 2 replies
- 1 kudos
- 7472 Views
- 2 replies
- 1 kudos
- 1 kudos
Bucketing is physical partition of the the table but the Zordering is arrangement of records in a file , in most optimal manner
- 1 kudos
- 2109 Views
- 1 replies
- 0 kudos
- 2109 Views
- 1 replies
- 0 kudos
- 0 kudos
There isn’t a problem purging old data. When using auto loader it’ll take into account new data being added.
- 0 kudos
- 1999 Views
- 1 replies
- 2 kudos
What is the best practice for generating jobs in an automated fashion?
What is the best practice for generating jobs in an automated fashion?
- 1999 Views
- 1 replies
- 2 kudos
- 2 kudos
There are several approaches here. You can write an automation script that programmatically accesses Databricks API’s to generate configured jobs. You can also utilize the Databricks Terraform provider. The benefit of the latter approach is that Terr...
- 2 kudos
- 1195 Views
- 1 replies
- 0 kudos
How can I reduce the risk of data exfiltration while using Databricks
How can I reduce the risk of data exfiltration while using Databricks
- 1195 Views
- 1 replies
- 0 kudos
- 0 kudos
Databricks enterprise security and admin features allow customers to deploy Databricks using their own managed VPC/ VNET. This enables them to have greater flexibility and control over the configuration of their deployment architectureFor Azure follo...
- 0 kudos
- 1868 Views
- 0 replies
- 0 kudos
- 1868 Views
- 0 replies
- 0 kudos
- 2079 Views
- 0 replies
- 0 kudos
Newline characters mess up the table records
When creating tables from text files containing newline characters in the middle of the lines, the table records will null column values because the newline characters in the middle of the lines break the lines into two different records and fill up ...
- 2079 Views
- 0 replies
- 0 kudos
- 1428 Views
- 0 replies
- 0 kudos
SQL notebooks not seeing the latest table schema / content.
When we run the sql statements "DROP TABLE .... CREATE TABLE" for the same table in multiple places (different notebooks, jobs, ...) some notebooks may not see the most recent schema / content.
- 1428 Views
- 0 replies
- 0 kudos
- 1662 Views
- 1 replies
- 0 kudos
How do you deploy Container Services to a Cluster Pool?
To speed up cluster provisioning in the case of use Container Services, how do you pre-install it to a pool.
- 1662 Views
- 1 replies
- 0 kudos
- 0 kudos
Using Pools APIs.
- 0 kudos
- 2285 Views
- 1 replies
- 0 kudos
how often should I vacuum my Delta table?
I would like to know how often do I need to vacuum my delta table to clean old files?
- 2285 Views
- 1 replies
- 0 kudos
- 0 kudos
The requirements for Vacuum will depend on your application needs and the rate of arrival of new data. Vacuuming removes old versions of data.If you need to be able to query earlier versions of data many months after the original ingest time, then i...
- 0 kudos
- 3521 Views
- 2 replies
- 0 kudos
how to partition my Delta table?
I would like to follow best practices to partition my Delta table. Should I partition by unique ID or date?
- 3521 Views
- 2 replies
- 0 kudos
- 0 kudos
Depending on the amount of data per partition - you may also want to consider partitioning by week, month or quarter.The partitioning decision is often tied to the tiering model of data storage. For a Bronze ingest layer, the optimal partitioning is ...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 572 | |
| 349 | |
| 287 |
