- 6178 Views
- 6 replies
- 2 kudos
Setting up my first DLT Pipeline with 3rd party JSON data
I'm getting an error when I try to create a DLT Pipeline from a bunch of third-party app-usage data we have. Here's the error message:Found invalid character(s) among ' ,;{}()\n\t=' in the column names of your schema. Please upgrade your Delta table ...
- 6178 Views
- 6 replies
- 2 kudos
- 2 kudos
I found this other forum thread that looks potentially useful, but I can’t figure out either how to translate it to SQL to handle JSON, nor how to get the pipeline I’m working with to interpret the Python. When I switch to Python, it complains about ...
- 2 kudos
- 5920 Views
- 4 replies
- 0 kudos
What determines whether an experiment's "Rename/Permissions/Delete" context menu is active or grayed out in the Experiments page?
I have a couple of experiments in the machine learning workspace. Some I want to delete since they are clutter/were created just to test out the platform. However I can't because the options to delete them is grayed out (see pictures below)I was wond...
- 5920 Views
- 4 replies
- 0 kudos
- 0 kudos
It looks like from the documentation the user should have the 'can Manage' permission:https://docs.databricks.com/security/access-control/workspace-acl.html#mlflow-experiment-permissions-1
- 0 kudos
- 10955 Views
- 5 replies
- 4 kudos
Send custom logs to AWS cloudwatch from Notebook
I would like to send some custom logs (in Python) from my Databricks notebook to AWS Cloudwatch. For example: df = spark.read.json(".......................")logger.info("Successfully ingested data from json")Has someone succeeded in doing this before...
- 10955 Views
- 5 replies
- 4 kudos
- 4 kudos
Hi, You can integrate, please refer: https://aws.amazon.com/blogs/mt/how-to-monitor-databricks-with-amazon-cloudwatch/ and also you can configure audit logging to S3 and redirect it to cloudwatch from AWS. , refer: https://aws.amazon.com/blogs/mt/how...
- 4 kudos
- 2551 Views
- 1 replies
- 2 kudos
Resolved! " Please migrate to `databricks_group_role` "
With Databricks Terraform connector version 1.2.0, I use the following to make the AWS instance profile available to all users in the workspace:// Create AWS instance profileresource "aws_iam_instance_profile" "this" { name = "${var.prefix}_instance_...
- 2551 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello, I opened a support case a couple of months ago specifically about this. The answer I got was"Terraform team will revert the deprecated resource and update the document accordingly.", which has not happened so far. Either provide documentation ...
- 2 kudos
- 4505 Views
- 3 replies
- 4 kudos
Resolved! Data size inflates massively while ingesting
GoalImport and consolidate GBs / TBs of local data in 20-mb chunk parquet files into Databricks / Delta lake / partitioned tables.What I've DoneI took a small subset of data, roughly 72.5 GB and ingested using streaming below. The data is already seq...
- 4505 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Erik Louie​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 4 kudos
- 13355 Views
- 5 replies
- 2 kudos
Resolved! Errors on Python API for DeltaTables on Databricks Spark 10.4 LST
Hi,I am using an (Azure) Databricks Compute cluster in a Jupyter notebook using the Databricks connect Python package. Our cluster is on Databrick runtime 10.4 LST and coherently I am using the databricks-connect==10.4.18.In my notebook I am able to ...
- 13355 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Maarten van Raaij​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....
- 2 kudos
- 2057 Views
- 1 replies
- 1 kudos
Resolved! Didn't receive my certification
I took my databrick certification exam on Feb-08-2023 however did'nt receive my certification untill now . Can i get help ?
- 2057 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Djoumoi Moinsalima​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training and our team will get back to you shortly.
- 1 kudos
- 2806 Views
- 1 replies
- 0 kudos
Unable to connect to Azure Cosmos DB Cassandra API table using Azure databricks job
Getting below error:Query [id = , runId = ] terminated with exception: Failed to open native connection to Cassandra at {<name>.cassandra.cosmosdb.azure.com:10350} :: Method com/microsoft/azure/cosmosdb/cassandra/CosmosDbConnectionFactory$.createSess...
- 2806 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, The error looks like there is an issue connecting to csms-ws-ddicsg-dev-001.cassandra.cosmosdb.azure.com:10350. Could you please reverify this in networking config? Also, it will be helpful if you raise an Azure case simultaneously to check the n...
- 0 kudos
- 4041 Views
- 5 replies
- 1 kudos
Resolved! Data Engineer Certificate and badge not received
I have cleared the certification exam on 26th January 2023 with 91.11%, but still haven't received the certificate. I had given the exam with a different mail ID but I'm not receiving any emails from Databricks to that mail ID.​I've also raised a tic...
- 4041 Views
- 5 replies
- 1 kudos
- 1 kudos
Hello team, I have answered via salesforce ticket and sent Naeem their sign on link and resent the badge/certificate email. Thanks!
- 1 kudos
- 9903 Views
- 8 replies
- 1 kudos
Mounting File Share in init script of cluster
we have a flow where we have to process chunk of files from file share. currently we are moving the files first to storage account and then post processing move files back to file share again. this is adding to the execution time for moving files bac...
- 9903 Views
- 8 replies
- 1 kudos
- 1 kudos
Hi @Rahul Samant is this issue solved Please help we are also facing same issues
- 1 kudos
- 7843 Views
- 6 replies
- 10 kudos
Disable dbutils suggestion
I would like to turn off or suppress this message which is returned from the dbutils library. %r files <- dbutils.fs.ls("/dbfs/tmp/") For prettier results from dbutils.fs.ls(<dir>), please use `%fs ls <dir>`How can I do this?
- 7843 Views
- 6 replies
- 10 kudos
- 10 kudos
Hi @James Smith​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 10 kudos
- 4363 Views
- 2 replies
- 0 kudos
Bugs with text query parameters?
Hi all,I have query with a handful of text query parameters, I need to use to insert data into a table from a dashboard e.g.INSERT INTO user_data (first_name, middle_name, last_name, city, country, zip_code) VALUES ('{{first_name}}', '{{middle_name}}...
- 4363 Views
- 2 replies
- 0 kudos
- 0 kudos
Sorry my client wants all 3 fields. The middle name has to be there. There are a few cases where people have just one name.
- 0 kudos
- 4874 Views
- 1 replies
- 6 kudos
Resolved! grant the access permissions for specific container and also for specific folder within container in Azure Blob storage
Hi,regarding permissions for Azure Storage.we have created the Storage account (blob storage) and within the account we are going to create many containers and in which container we are going to have multiple folders and files.we want to grant permis...
- 4874 Views
- 1 replies
- 6 kudos
- 6 kudos
Hi @KVNARK .​ ,You can use the service principle in the azure active directory to grant specific access to that app and use that app credentials to create a new mount point.That will help you to give specific storage permission to users.
- 6 kudos
- 4099 Views
- 2 replies
- 4 kudos
what is the supported mssql connector for Databricks runtime 11.3LTS Scala 2.12 Spark 3.3.0?
We were using mssql connector -com.microsoft.azure:spark-mssql-connector_2.12_3.0:1.0.0-alpha with 10.3LTS DBR. As we need to upgrade to higher version of DBR to make use of new functions like unpivot/melt in the notebooks. -com.microsoft.azure:spark...
- 4099 Views
- 2 replies
- 4 kudos
- 4 kudos
Is the spark 3.3 series even supported by the connector yet ?As per the [github link](https://github.com/microsoft/sql-spark-connector#current-releases) - assuming this is the library you are trying to use ?The latest Spark 2.4.x compatible connector...
- 4 kudos
- 2419 Views
- 1 replies
- 0 kudos
Running stateful spark streaming example fails https://www.databricks.com/blog/2022/10/18/python-arbitrary-stateful-processing-structured-streaming.html
ERROR:py4j.clientserver:There was an exception while executing the Python Proxy on the Python Side. Traceback (most recent call last): File "/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/clientserver.py", line 617, in _call_proxy retu...
- 2419 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, The error looks like the failure was fetched from the PY configuration? Could you please provide the whole snippet of the error?
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
