- 27924 Views
- 17 replies
- 12 kudos
How to instantiate Databricks spark context in a python script?
I want to run a block of code in a script and not in a notebook on databricks, however I cannot properly instantiate the spark context without some error.I have tried ` SparkContext.getOrCreate()`, but this does not work.Is there a simple way to do t...
- 27924 Views
- 17 replies
- 12 kudos
- 12 kudos
Is there some solution for this.We got struck where a cluster having unity catalog is not able to get spark context.This is not allowing to use distributed nature of spark in databricks.
- 12 kudos
- 4473 Views
- 2 replies
- 0 kudos
Delta Live Tables INSUFFICIENT_PERMISSIONS
I have a delta live table pipeline which reads from a delta table then applies 3 layers of transformations before merging the legs of the pipeline and outputting. I am getting this error when I run my pipeline against unity catalog.```org.apache.spar...
- 4473 Views
- 2 replies
- 0 kudos
- 0 kudos
I don't see how that can be the underlying issue because. 1. the first step of the pipeline which reads from unity catalog and azure sql is just fine.2. when I remove the enrichment logic from the second step of the and just pass the table input as i...
- 0 kudos
- 13380 Views
- 7 replies
- 3 kudos
Resolved! Changing shuffle.partitions with spark.conf in a spark stream - isn't respected even after a checkpoint
Question about spark checkpoints and offsets in a running streamWhen the stream started I needed tons of partitions, so we've set it with spark.conf to 5000As expected offsets in the checkpoint contain this info and the job used this value. Then we'...
- 13380 Views
- 7 replies
- 3 kudos
- 3 kudos
@Jose Gonzalez​ thanks for that information! This is super useful. I was struggling why my streaming still using 200 partitions. This is quite a paint for me because changing checkpoint will insert all data from the source. Do you know where this can...
- 3 kudos
- 3885 Views
- 2 replies
- 2 kudos
Resolved! Serverless Compute - How to determine if being used programatically
Hi, We use a common notebook for all our "common" settings, this notebook is called in the first cell of each notebook we develop. This issue we are now having is that we need 2 common notebooks, one for a normal shared compute and one for serverle...
- 3885 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Dave1967 ,If you know any spark config command that is not supported in serverless, then build your logic around this command using try, catch:def is_config_supported(): try: spark.sparkContext.getConf() return True except...
- 2 kudos
- 1662 Views
- 1 replies
- 1 kudos
Assistance Required: Issues Creating External Table in Legacy Hive Metastore
I am currently trying to create an external table with an external location in the legacy Hive metastore. As part of this process, I have also created the necessary secret scope using below steps, but I am still encountering issues when attempting to...
- 1662 Views
- 1 replies
- 1 kudos
- 1 kudos
Did you mount the Location using the secrets? https://learn.microsoft.com/en-us/azure/databricks/dbfs/mounts#mount-adls-gen2-or-blob-storage-with-abfs
- 1 kudos
- 2611 Views
- 3 replies
- 0 kudos
Shared access mode and py4j.security.Py4JSecurityException
Hi,We are getting below exception on shared access mode cluster.py4j.security.Py4JSecurityException: Method public java.lang.String com.databricks.backend.common.rpc.CommandContext.toJson() is not whitelisted on class class com.databricks.backend.com...
- 2611 Views
- 3 replies
- 0 kudos
- 0 kudos
@gchandra I'm aware that I can use .safeToJson() but did you see the code of py4j.security.WhitelistingPy4JSecurityManager ? How it works what can be whitelist what can't be whitelist ? I don't see py4j.security.WhitelistingPy4JSecurityManager on o...
- 0 kudos
- 7261 Views
- 6 replies
- 2 kudos
Resolved! What would happen it my dynamic IP changed in IP Access list?
On this feature IP access lists IP access lists - Azure Databricks | Microsoft Docs, what we observe is that if your IP is not on the access list, you cannot modify the list via API since you are not on trusted location. What if I specify only 1 IP s...
- 7261 Views
- 6 replies
- 2 kudos
- 2 kudos
Curious to learn if somebody also figured out a way to solve for the above as we've encountered this situation and are now locked out...
- 2 kudos
- 33807 Views
- 4 replies
- 0 kudos
Select job cluster vs all purpose cluster
I have workflow and need to run at every 1 minute interval, it is rest api call, should I go for all purpose cluster or job cluster to meet the SLA. We need to get the as soon as it is available.
- 33807 Views
- 4 replies
- 0 kudos
- 0 kudos
@greyamber Interactive cluster costs two time more than job cluster. can you explain use-case of why job API needs to invoked and what API is doing.
- 0 kudos
- 1534 Views
- 2 replies
- 0 kudos
Extract DLT Pipeline Logs to a delta table
I want to export the dlt pipeline run details into a delta table . I want a table that should have data like this.
- 1534 Views
- 2 replies
- 0 kudos
- 0 kudos
Is there any way i can use event hook to log into my own delta table. If anyone have working example then that would be great.
- 0 kudos
- 1949 Views
- 2 replies
- 0 kudos
VS Code Databricks Connect Cluster Configuration
I am currently setting up the VSCode extension for Databricks Connect, and it’s working fine so far. However, I have a question about cluster configurations. I want to access Unity Catalog from VSCode through the extension, and I’ve noticed that I ca...
- 1949 Views
- 2 replies
- 0 kudos
- 1457 Views
- 1 replies
- 0 kudos
# Databricks notebook source throws FileNotFoundError: [Errno 2] No such file with PyCharm plugin
I started using Databricks plugin in Pycharm.If I have first line in my code:# Databricks notebook source ...Running such a notebook with a plugin on cluster fails with the message:FileNotFoundError: [Errno 2] No such file or directory: '/Workspace/U...
- 1457 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @aleknandrius,How are you doing today?As per my understanding, It seems that the # Databricks notebook source line is causing confusion when running your PyCharm code on Databricks. This line is usually added to identify notebook cells, but in a P...
- 0 kudos
- 25046 Views
- 7 replies
- 6 kudos
Resolved! Delta tables: Cannot set default column mapping mode to "name" in Python for delta tables
Hello,I am trying to write Delta files for some CSV data. When I docsv_dataframe.write.format("delta").save("/path/to/table.delta")I get: AnalysisException: Found invalid character(s) among " ,;{}()\n\t=" in the column names of yourschema.Having look...
- 25046 Views
- 7 replies
- 6 kudos
- 6 kudos
I still get the error when I try any method. The column names with spaces are throwing error [DELTA_INVALID_CHARACTERS_IN_COLUMN_NAMES] Found invalid character(s) among ' ,;{}()\n\t=' in the column names of your schema.df1.write.format("delta") \ .mo...
- 6 kudos
- 5097 Views
- 5 replies
- 0 kudos
Web Terminal
How can I use web terminal within my azure databricks workspace if the workspace is provisioned with a private end points i.e. Allow Public Network Access is disabled.I have tried accessing web terminal from Apps tab and Bottom panel of a notebook.Th...
- 5097 Views
- 5 replies
- 0 kudos
- 0 kudos
Gotcha. See whether this link is helpful. https://learn.microsoft.com/en-us/azure/databricks/connect/storage/tutorial-azure-storage#grant-your-azure-databricks-workspace-access-to-azure-data-lake-storage-gen2
- 0 kudos
- 14525 Views
- 2 replies
- 3 kudos
Partition in Spark
Hi Community, Need your help on understanding below topics.. I have a huge transaction file ( 20GB ) partition by transaction_date column , parquet file. I have evenly distributed data ( no skew ). There are 10 days of data and we have 10 partition f...
- 14525 Views
- 2 replies
- 3 kudos
- 3 kudos
I read a .zip file in Spark and get unreadable data when I run show() on the data frame.When I check the number of partitions using df.rdd.getNumPartitions(), I get 8 (the number of cores I am using). Shouldn't the partition count be just 1 as I read...
- 3 kudos
- 11738 Views
- 10 replies
- 3 kudos
Azure Databricks Workflows with Git Integration
Hello,I receive a very weird error when attempting to connect my workflows tasks to a remote git on azure devops.As per documentation: "For a Git repository, the path relative to the repository root."Then, I use directly the name of the notebook file...
- 11738 Views
- 10 replies
- 3 kudos
- 3 kudos
Hi Federico, The error in Error 1.png didn't look right. Since you already selected the git source for the job, you should be able to use a relative path. If you continue to run into this issue, can you please submit a support ticket if you have a Su...
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
