- 1026 Views
- 0 replies
- 0 kudos
- 1026 Views
- 0 replies
- 0 kudos
Forum Posts
- 1791 Views
- 1 replies
- 1 kudos
- 1791 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
The open source spark connector for Snowflake is available by default in the Databricks runtime. To connect you can use the following code: # Use secrets DBUtil to get Snowflake credentials. user = dbutils.secrets.get("<scope>", "<secret key>") passw...
- 1 kudos
- 2140 Views
- 1 replies
- 0 kudos
How to Prevent Duplicate Entries to enter to delta lake of Azure Storage?
I Have a Dataframe stored in the format of delta into Adls, now when im trying to append new updated rows to that delta lake it should, Is there any way where i can delete the old existing record in delta and add the new updated Record.There is a uni...
- 2140 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
To achieve this you should use a merge command that will update rows that are existing with the unique ID. This will update the rows that already exist and insert the rows that do not. If you want to do it manually, you could delete rows using the DE...
- 0 kudos
- 1328 Views
- 0 replies
- 1 kudos
How to get access to Databricks SQL analytics?
am trying to do this tutorial about databricks sql analytics (https://docs.microsoft.com/en-us/azure/databricks/sql/get-started/admin-quickstart) but when i create my databricks workspace i do not have the icon at the bottom of the sidebar to access ...
- 1328 Views
- 0 replies
- 1 kudos
- 1060 Views
- 0 replies
- 0 kudos
- 1060 Views
- 0 replies
- 0 kudos
- 1604 Views
- 0 replies
- 0 kudos
How to delete users by email id using azure SCIM api in databricks?
I need to know if there is a way to delete a user from databricks using email only using SCIM api? As of now I can see it can only delete user by ID which means I need to first retrive the ID of the user and then use it to delete.I am using this api ...
- 1604 Views
- 0 replies
- 0 kudos
- 5880 Views
- 1 replies
- 0 kudos
Resolved! Do ganglia report incorrect memory stats?
I am looking at the memory utilization of the executors and I see the heap utilization of the executor is far less than what is reported in the Ganglia. Why do ganglia report incorrect memory details.
- 5880 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Ganglia reports the memory utilization at the system level. Say for example if the JVM has Xmx value of 100 GB. At some point, it will occupy 100GB and then with a Garbage collection, it will clear off the heap. Once the GC frees up the memory, th...
- 0 kudos
- 1260 Views
- 0 replies
- 0 kudos
- 1260 Views
- 0 replies
- 0 kudos
- 1401 Views
- 0 replies
- 0 kudos
Where is the root Azure Storage instance?
I am trying to access logs from my Databricks notebook which is run as a job. I would like to see these logs in an azure storage account.
- 1401 Views
- 0 replies
- 0 kudos
- 2249 Views
- 0 replies
- 1 kudos
What is the most efficient way to read in a partitioned parquet file with pyspark?
I work with parquet files stored in AWS S3 buckets. They are multiple TB in size and partitioned by a numeric column containing integer values between 1 and 200, call it my_partition. I read in and perform compute actions on this data in Databricks w...
- 2249 Views
- 0 replies
- 1 kudos
- 3333 Views
- 1 replies
- 0 kudos
Resolved! Is it mandatory to checkpoint my streaming query.
I have ad-hoc one-time streaming queries where I believe checkpoint won't give any value add. Should I still use checkpointing
- 3333 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
It's not mandatory. But the strong recommendation is to use Checkpointing for Streaming irrespective of your use case. This is because the default checkpoint location can get a lot of files over time as there is no graceful guaranteed cleaning in pla...
- 0 kudos
- 3956 Views
- 2 replies
- 0 kudos
Should/Can I use spark streaming for Batch workloads?
Its preferable to use spark streaming (with Delta) for batch workloads rather then regular batch. With the trigger.once trigger whenever the streaming job is started it will process whatever is available in the source (kafka/kinesis/File System) and ...
- 3956 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
The streaming checkpoint mechanism is independent of the Trigger type. The way checkpoint works are it creates an offset file when processing the batch and once the batch is completed it creates a commit file for that batch in the checkpoint director...
- 0 kudos
- 1302 Views
- 1 replies
- 0 kudos
How to migrate to Auto-loader without downtime?
I have an S3-SQS workload. Is it possible to migrate the workload to autoloader without downtime? What are the migration guidelines.
- 1302 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
The SQS queue used by the existing application can be utilized by the auto-loader thereby ensuring minimal downtime
- 0 kudos
- 2178 Views
- 1 replies
- 0 kudos
Resolved! Why do I always get an error on querying the Parquet table - Parquet does not support timestamp
- 2178 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
The issue can happen if the Hive syntax for table creation is used instead of the Spark syntax. Read more here: https://docs.databricks.com/spark/latest/spark-sql/language-manual/sql-ref-syntax-ddl-create-table-hiveformat.htmlThe issue mentioned in t...
- 0 kudos
- 6828 Views
- 1 replies
- 0 kudos
Resolved! How to track the history of schema changes for a Delta table
I have a Delta table that had schema changes in multiple commits. I wanted to track all these schema changes that happened on the Delta table. The "DESCRIBE HISTORY" is not useful as it logs the schema change made by ALTER TABLE operations.
- 6828 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
When a write operation is performed with columns added. we are not explicitly showing that in DESCRIBE HISTORY output. Only an entry is made for write. and in the operation Parameters, it's not showing anything about schema evolution. whereas if we d...
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 483 | |
| 349 | |
| 287 |
