- 12935 Views
- 3 replies
- 3 kudos
Resolved! Differences between Spark Cluster Manager and Databricks Cluster Manager?
I didn't found any documentation on Databricks Cluster Manager. Could anyone give me some resources on this topic?
- 12935 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @John William​ Databricks clusters use Spark's Standalone cluster manager. Each Databricks cluster has its own standalone Master and Worker processes run inside of the LXC containers and share a lifecycle with the cluster. Each cluster has a singl...
- 3 kudos
- 4593 Views
- 4 replies
- 11 kudos
Resolved! Databricks pyspark - Find columns in xls file.
Hello everyone, every day I extract data into xls files but the column position changes every day. Is there any way to find these columns within the file?Here's a snippet of my code.df = spark.read.format("com.crealytics.spark.excel")\ .option("hea...
- 4593 Views
- 4 replies
- 11 kudos
- 11 kudos
Hi @welder martins​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
- 11 kudos
- 3071 Views
- 2 replies
- 3 kudos
DLT Event Log
I am trying to utilize the Event Log DLT is keeping updated, I noticed some of the fields are consistently empty/null.In the Event Log, located ".../storage/system/events", I see the field "origin" and there are nested fields within which are empty/n...
- 3071 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Kristian Foster​,The following docs will provide more details on the event log schema. Please refer to this link https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-event-log.html#monitor-pipelines-with-the-delta-live-tables...
- 3 kudos
- 2584 Views
- 2 replies
- 2 kudos
What is the long-term plan for the official Databricks Docker Containers?
Hi, I'd like to ask you, how much resources do you plan to dedicate to the maintenance/development of the official Databricks Docker images, please? Do you have a view on the longer-term plan for these docker images? It seems to be maintained, but i...
- 2584 Views
- 2 replies
- 2 kudos
- 8621 Views
- 6 replies
- 3 kudos
GRANT permission not working on Storage Credential using DBR 11.2 Shared mode
We were trying to update permissions of Storage credential using DBR 11.2 Shared mode but running into following issue Operation not allowed: GRANT(line 1, pos 0) PFA complete error in file.Is the above issue with cluster permission or changing permi...
- 8621 Views
- 6 replies
- 3 kudos
- 3 kudos
Hi @Parth Salvi​ , could you please paste the full error here, also, it is better to create a support case with Databricks with the whole error.
- 3 kudos
- 7206 Views
- 3 replies
- 0 kudos
XSS.svg#.png
"><script>alert(1)</script>
- 7206 Views
- 3 replies
- 0 kudos
- 0 kudos
"><iframe onLoad iframe onLoad="javascript:javascript:alert(1)"></iframe onLoad>xss
- 0 kudos
- 8192 Views
- 3 replies
- 7 kudos
Resolved! Hands-On exercise material
Hi FriendsI am following Databricks Customer Academy training material, and created a Databricks service in Azure Trial account and was able to launched a single node cluster there. Could you please guide me on how to do all the hands-on?
- 8192 Views
- 3 replies
- 7 kudos
- 7 kudos
Thanks Doug, I was able to locate .dbc file, appreciate your response. Best RegardsAnindya
- 7 kudos
- 27978 Views
- 0 replies
- 0 kudos
Max file size allowed to import into Databricks community edition ?
Hi All,I have few questions using the community edition 1) max file size that is allowed to be uploaded (data file) in community edition ?2) is XML file supported as well ? Regards,Rakesh
- 27978 Views
- 0 replies
- 0 kudos
- 1639 Views
- 0 replies
- 2 kudos
Is it possible to connect to IPython Kernel from local or client outside databricks cluster?
When looking in the standard output of a notebook run in a cluster, we get this message: "To connect another client to this kernel, use: /databricks/kernel-connections-dj8dj93d3d3.json"Is it possible to connect to the databricks ipython kernel and ma...
- 1639 Views
- 0 replies
- 2 kudos
- 11450 Views
- 6 replies
- 5 kudos
Resolved! COPY INTO generating duplicate rows in Delta table
Hello Everyone,I'm trying to bulk load tables from a SQL server database into ADLS as parquet files and then loading these files into Delta tables (raw/bronze). I had done a one off history/base load but my subsequent incremental loads (which had a d...
- 11450 Views
- 6 replies
- 5 kudos
- 8277 Views
- 4 replies
- 16 kudos
Resolved! Rounding in the databricks SQL editor
In SQL editor (inside SQL experience), executing expressions 1 – 0.999 gives the result 0.
- 8277 Views
- 4 replies
- 16 kudos
- 12802 Views
- 2 replies
- 3 kudos
How to Prevent Duplicate Entries to enter to delta lake of Azure Storage
I Have a Dataframe stored in the format of delta into Adls, now when im trying to append new updated rows to that delta lake it should, Is there any way where i can delete the old existing record in delta and add the new updated Record.There is a uni...
- 12802 Views
- 2 replies
- 3 kudos
- 3 kudos
You should use a MERGE command on this table to match records on the unique column. Delta Lake does not enforce primary keys so if you append only the duplicate ids will appear. Merge will provide you the functionality you desire. https://docs.databr...
- 3 kudos
- 7097 Views
- 4 replies
- 5 kudos
Resolved! How do I remove a filter from the visualisations pane in databricks sql, 2 i am able to remove but for some reason the Year filter doesn't have a delete option? The documentation isn't very comprehensive.
I want to remove one of these filters but there doesn't seem to be a way to do it.
- 7097 Views
- 4 replies
- 5 kudos
- 5 kudos
We're always here, even for newbie errors @Vanessa Van Gelder​ !Thanks for posting, and thanks @Hubert Dudek​ for always being so helpful.
- 5 kudos
- 10874 Views
- 0 replies
- 4 kudos
AttributeError: 'list' object has no attribute 'columns' - PySpark
Hi All,i am getting the below error when i am ingesting the data from source file , source file is also attached , i have tried in both Community edition and Azure databricks as well getting the same error , can any one suggest me the solution ? # ...
- 10874 Views
- 0 replies
- 4 kudos
- 6247 Views
- 1 replies
- 2 kudos
Resolved! unable to replace null with 0 in dataframe using Pyspark databricks notebook (community edition)
Hello Experts,I am unable to replace nulls with 0 in a dataframe ,please refer to the screen shotfrom pyspark.sql.functions import col emp_csv_df = emp_csv_df.na.fill(0).withColumn("Total_Sal",col('sal')+col('comm')) display(emp_csv_df)erorr desired ...
- 6247 Views
- 1 replies
- 2 kudos
- 2 kudos
I bet that it is not real null but the string "null". Please check what is in the source and try luck with replacing it.
- 2 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
