- 3609 Views
- 2 replies
- 3 kudos
Equivalent Machine Types between Databricks on Azure and GCP
Hi All,Hope everyone is doing well.We are currently validating Databricks on GCP and Azure.We have a python notebook that does some ETL (Copy, extract zip files and process files within the zip files)Our Cluster Config on AzureDBX Runtime - 10.4 - Dr...
- 3609 Views
- 2 replies
- 3 kudos
- 3 kudos
hi @Tunde Abib​ , I have gone through the links while updating, but did not see any major documented slow downs mentioned in them.
- 3 kudos
- 3905 Views
- 6 replies
- 5 kudos
KB Feedback Discussion  In addition to the Databricks Community, we have a Support team that maintains a Knowledge Base (KB). The KB contains answers ...
KB Feedback Discussion In addition to the Databricks Community, we have a Support team that maintains a Knowledge Base (KB). The KB contains answers to common questions about Databricks, as well as information on optimisation and troubleshooting.Thes...
- 3905 Views
- 6 replies
- 5 kudos
- 9080 Views
- 5 replies
- 7 kudos
Resolved! What's the easiest way to upsert data into a table? (Azure ADLS Gen2)
I had been trying to upsert rows into a table in Azure Blob Storage (ADLS Gen 2) based on two partitions (sample code below). insert overwrite table new_clicks_table partition(client_id, mm_date) select click_id ,user_id ,click_timestamp_gmt ,ca...
- 9080 Views
- 5 replies
- 7 kudos
- 7 kudos
Below code might help youPython- (df.write .mode("overwrite") .option("partitionOverwriteMode", "dynamic") .saveAsTable("default.people10m") ) SQL- SET spark.sql.sources.partitionOverwriteMode=dynamic; INSERT OVERWRITE TABLE default.people10m...
- 7 kudos
- 14713 Views
- 11 replies
- 8 kudos
Resolved! Databricks lakehouse platform administration accreditation
How to complete the Databricks lakehouse platform administration for free just like Lakehouse fundamentals. How to get the accreditation for platform administrator like lakehouse fundamentals.
- 14713 Views
- 11 replies
- 8 kudos
- 1794 Views
- 1 replies
- 7 kudos
Regarding my free lake house 100 points
hi @Christy Seto​, i have cleared the lake house exam before 30 november 2022 and was eligible to get a 100 community points , i have cleared with the email id of manpreet.kaur@celebaltech.com but till now i havent get 100 points . i have edited my e...
- 1794 Views
- 1 replies
- 7 kudos
- 7 kudos
Hi @Rishabh Pandey​ Please raise a request via this link this might help you.
- 7 kudos
- 3008 Views
- 1 replies
- 1 kudos
Resolved! Modify the Json Schema Stored in a File for AutoLoader
We are reading over an S3 bucket which contains a several million json files. The schema from the read is stored in a json file in the dbfs filestore. This file is then utilized by autoloader to write new files nightly to a delta table. The schema is...
- 3008 Views
- 1 replies
- 1 kudos
- 1 kudos
if anyone is curious I ended up just passing the schema as a string to .schema(eval(the_schema)) in StructType format and not using the file based approach.
- 1 kudos
- 1360 Views
- 0 replies
- 0 kudos
.Net thrift client for sql warehouse
I'm trying to port python-sql thrift client to .net and I receive a 500 error when trying to open a session.Is there a way to have an sql warehouse server mock in order to investigate the error.
- 1360 Views
- 0 replies
- 0 kudos
- 3147 Views
- 3 replies
- 0 kudos
- 3147 Views
- 3 replies
- 0 kudos
- 0 kudos
there is no single answer to this.If you look at parquet, which is a very common format on data lakes:https://parquet.apache.org/docs/file-format/nulls/and on SO
- 0 kudos
- 10264 Views
- 2 replies
- 1 kudos
Unable to save CSV file into DBFS
Hello,I have took the azure datasets that are available for practice. I got the 10 days data from that dataset and now I want to save this data into DBFS in csv format. I have facing an error :" No such file or directory: '/dbfs/tmp/myfolder/mytest.c...
- 10264 Views
- 2 replies
- 1 kudos
- 1 kudos
You can use spark dataframe to read and write the CSV files-Read- df=spark.read.csv("Path") Write- df.write.csv("Path")
- 1 kudos
- 2118 Views
- 2 replies
- 4 kudos
How much time does it take for the databricks partner account to get created
How much time does it take for the databricks partner account to get created after we submit the application to databricks.?
- 2118 Views
- 2 replies
- 4 kudos
- 12456 Views
- 10 replies
- 6 kudos
Resolved! onsite exam center registration Databricks Certified Associate Developer for Apache Spark 3
Dear All @Nadia Elsayed​ @Vidula Khanna​ @Harshjot Singh​ @Jose Gonzalez​ @Joseph Kambourakis​ Hope you are well and had a good weekend.I am still waiting to receive voucher after redeeming points which is due this weekMy issue is slots are full to ...
- 12456 Views
- 10 replies
- 6 kudos
- 6 kudos
I have the same problem as you. I submitted a ticket to Databricks "Help to re-schedule assessment day in webassessor", but they responsed as below: " Please accept my apologies for the inconvenience caused and the delay in responding. I'm sorry to i...
- 6 kudos
- 2099 Views
- 0 replies
- 0 kudos
Overwrite still saves numerous parquet files in storage container
I inherited this environment and my question is we have a job that mines the the data lake and creates a table that's is grouped by unit number and their data points. The job runs every 10 minutes. We then connect to that table direct query power bi ...
- 2099 Views
- 0 replies
- 0 kudos
- 8160 Views
- 3 replies
- 7 kudos
Resolved! How nulls are stored in delta lake and databricks?
In my findings I have found a lot of delta tables in the lake house to be sparse so just wondering what space data lake takes to store null data and also any suggestions to handle sparse data tables in lake house would be appreciated.I also want to o...
- 8160 Views
- 3 replies
- 7 kudos
- 7 kudos
As delta uses parquet files to store data inside delta:"Nullity is encoded in the definition levels (which is run-length encoded). NULL values are not encoded in the data. For example, in a non-nested schema, a column with 1000 NULLs would be encoded...
- 7 kudos
- 8654 Views
- 4 replies
- 5 kudos
How to %run a list of notebooks in Databricks
I'd like to %run a list of notebooks from another Databricks notebook.my_notebooks = ["./setup", "./do_the_main_thing", "./check_results"] for notebook in my_notebooks: %run notebookThis doesn't work ofcourse. I don't want to use dbutils.notebook....
- 8654 Views
- 4 replies
- 5 kudos
- 5 kudos
Please refer below codeimport scala.concurrent.{Future, Await} import scala.concurrent.duration._ import scala.util.control.NonFatal case class NotebookData(path: String, timeout: Int, parameters: Map[String, String] = Map.empty[String, String]) ...
- 5 kudos
- 9597 Views
- 2 replies
- 2 kudos
Resolved! How to get the count of files/partition for a Delta table?
I have a delta table and I run optimize command regularly. However, I still see a large number of files in the table. I wanted to get a break up of the files in each partition and identify which partition has more files. What is the easiest way to ge...
- 9597 Views
- 2 replies
- 2 kudos
- 2 kudos
The below code snippet will give details about the file count per partitionimport com.databricks.sql.transaction.tahoe.DeltaLog import org.apache.hadoop.fs.Path val deltaPath = "<table_path>" val deltaLog = DeltaLog(spark, new Path(deltaPath + "/_d...
- 2 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
