- 1638 Views
- 1 replies
- 1 kudos
Assistance Required: Issues Creating External Table in Legacy Hive Metastore
I am currently trying to create an external table with an external location in the legacy Hive metastore. As part of this process, I have also created the necessary secret scope using below steps, but I am still encountering issues when attempting to...
- 1638 Views
- 1 replies
- 1 kudos
- 1 kudos
Did you mount the Location using the secrets? https://learn.microsoft.com/en-us/azure/databricks/dbfs/mounts#mount-adls-gen2-or-blob-storage-with-abfs
- 1 kudos
- 2542 Views
- 3 replies
- 0 kudos
Shared access mode and py4j.security.Py4JSecurityException
Hi,We are getting below exception on shared access mode cluster.py4j.security.Py4JSecurityException: Method public java.lang.String com.databricks.backend.common.rpc.CommandContext.toJson() is not whitelisted on class class com.databricks.backend.com...
- 2542 Views
- 3 replies
- 0 kudos
- 0 kudos
@gchandra I'm aware that I can use .safeToJson() but did you see the code of py4j.security.WhitelistingPy4JSecurityManager ? How it works what can be whitelist what can't be whitelist ? I don't see py4j.security.WhitelistingPy4JSecurityManager on o...
- 0 kudos
- 7138 Views
- 6 replies
- 2 kudos
Resolved! What would happen it my dynamic IP changed in IP Access list?
On this feature IP access lists IP access lists - Azure Databricks | Microsoft Docs, what we observe is that if your IP is not on the access list, you cannot modify the list via API since you are not on trusted location. What if I specify only 1 IP s...
- 7138 Views
- 6 replies
- 2 kudos
- 2 kudos
Curious to learn if somebody also figured out a way to solve for the above as we've encountered this situation and are now locked out...
- 2 kudos
- 33704 Views
- 4 replies
- 0 kudos
Select job cluster vs all purpose cluster
I have workflow and need to run at every 1 minute interval, it is rest api call, should I go for all purpose cluster or job cluster to meet the SLA. We need to get the as soon as it is available.
- 33704 Views
- 4 replies
- 0 kudos
- 0 kudos
@greyamber Interactive cluster costs two time more than job cluster. can you explain use-case of why job API needs to invoked and what API is doing.
- 0 kudos
- 1486 Views
- 2 replies
- 0 kudos
Extract DLT Pipeline Logs to a delta table
I want to export the dlt pipeline run details into a delta table . I want a table that should have data like this.
- 1486 Views
- 2 replies
- 0 kudos
- 0 kudos
Is there any way i can use event hook to log into my own delta table. If anyone have working example then that would be great.
- 0 kudos
- 1883 Views
- 2 replies
- 0 kudos
VS Code Databricks Connect Cluster Configuration
I am currently setting up the VSCode extension for Databricks Connect, and it’s working fine so far. However, I have a question about cluster configurations. I want to access Unity Catalog from VSCode through the extension, and I’ve noticed that I ca...
- 1883 Views
- 2 replies
- 0 kudos
- 1420 Views
- 1 replies
- 0 kudos
# Databricks notebook source throws FileNotFoundError: [Errno 2] No such file with PyCharm plugin
I started using Databricks plugin in Pycharm.If I have first line in my code:# Databricks notebook source ...Running such a notebook with a plugin on cluster fails with the message:FileNotFoundError: [Errno 2] No such file or directory: '/Workspace/U...
- 1420 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @aleknandrius,How are you doing today?As per my understanding, It seems that the # Databricks notebook source line is causing confusion when running your PyCharm code on Databricks. This line is usually added to identify notebook cells, but in a P...
- 0 kudos
- 24738 Views
- 7 replies
- 6 kudos
Resolved! Delta tables: Cannot set default column mapping mode to "name" in Python for delta tables
Hello,I am trying to write Delta files for some CSV data. When I docsv_dataframe.write.format("delta").save("/path/to/table.delta")I get: AnalysisException: Found invalid character(s) among " ,;{}()\n\t=" in the column names of yourschema.Having look...
- 24738 Views
- 7 replies
- 6 kudos
- 6 kudos
I still get the error when I try any method. The column names with spaces are throwing error [DELTA_INVALID_CHARACTERS_IN_COLUMN_NAMES] Found invalid character(s) among ' ,;{}()\n\t=' in the column names of your schema.df1.write.format("delta") \ .mo...
- 6 kudos
- 4971 Views
- 5 replies
- 0 kudos
Web Terminal
How can I use web terminal within my azure databricks workspace if the workspace is provisioned with a private end points i.e. Allow Public Network Access is disabled.I have tried accessing web terminal from Apps tab and Bottom panel of a notebook.Th...
- 4971 Views
- 5 replies
- 0 kudos
- 0 kudos
Gotcha. See whether this link is helpful. https://learn.microsoft.com/en-us/azure/databricks/connect/storage/tutorial-azure-storage#grant-your-azure-databricks-workspace-access-to-azure-data-lake-storage-gen2
- 0 kudos
- 14432 Views
- 2 replies
- 3 kudos
Partition in Spark
Hi Community, Need your help on understanding below topics.. I have a huge transaction file ( 20GB ) partition by transaction_date column , parquet file. I have evenly distributed data ( no skew ). There are 10 days of data and we have 10 partition f...
- 14432 Views
- 2 replies
- 3 kudos
- 3 kudos
I read a .zip file in Spark and get unreadable data when I run show() on the data frame.When I check the number of partitions using df.rdd.getNumPartitions(), I get 8 (the number of cores I am using). Shouldn't the partition count be just 1 as I read...
- 3 kudos
- 11425 Views
- 10 replies
- 3 kudos
Azure Databricks Workflows with Git Integration
Hello,I receive a very weird error when attempting to connect my workflows tasks to a remote git on azure devops.As per documentation: "For a Git repository, the path relative to the repository root."Then, I use directly the name of the notebook file...
- 11425 Views
- 10 replies
- 3 kudos
- 3 kudos
Hi Federico, The error in Error 1.png didn't look right. Since you already selected the git source for the job, you should be able to use a relative path. If you continue to run into this issue, can you please submit a support ticket if you have a Su...
- 3 kudos
- 2888 Views
- 2 replies
- 0 kudos
Updating records with auto loader
I want to ingest JSON files from an S3 bucket into a Databricks table using an autoloader.A job runs every few hours to write the combined JSON data to the table.Some records might be updates to existing records, identifiable by a specific key.I want...
- 2888 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @databrickser ,In theory, it is possible to use Auto Loader with foreachBatch to update existing records.The below code snippet show a working solution:from datetime import datetime from pyspark.sql import DataFrame from pyspark.sql.functions impo...
- 0 kudos
- 1537 Views
- 2 replies
- 0 kudos
Single task job that runs SQL notebook, can't retrieve results
Hello,We are integrating databricks and I need to run a job with single task that will run notebok with SQL query in it. I Can only use SQL warehouse and no cluster, I need to retrieve a result of the the notebook task but I can't see the results. Is...
- 1537 Views
- 2 replies
- 0 kudos
- 0 kudos
> I need to retrieve a result of the the notebook taskIf you want to know if the task run has succeeded or not, you can enable the "lakeflow" system schema and you'll find the logs of jobs and task runs.You could then use the above info to execute a...
- 0 kudos
- 1000 Views
- 1 replies
- 0 kudos
AWS Role of NACL outbound 3306 port
When using databricks for AWSI need to open 3306 to the NACL outbound port of the subnet where the endpoint is locatedI understand this is to communicate with the meta store of Databricks on the instanceAm I right to understand?If not, please let me ...
- 1000 Views
- 1 replies
- 0 kudos
- 0 kudos
You are correct, the intention of this port is to connect to the Hive metastore
- 0 kudos
- 2021 Views
- 1 replies
- 0 kudos
Asset Bundles from Workspace for CI/CD
Hello there,I am exploring the possibilities for CI/CD from a DEV-Workspace to PROD. Besides the Notebooks (which can easily be handled by the GIT-provider) I am mainly interested in the Deployment o Jobs/Clusters/DDl...I can nowhere find a tutorial ...
- 2021 Views
- 1 replies
- 0 kudos
- 0 kudos
i think the dab mlop stack template is pretty helpful re: how to bundle, schedule and trigger custom jobshttps://docs.databricks.com/en/dev-tools/bundles/mlops-stacks.htmlyou can bundle init it locally and it should give you the skeleton of how to bu...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
