- 3355 Views
- 4 replies
- 1 kudos
Need help to understand Databricks workspace service principle token expire calculate
Hi Team, Need assistance to understand Databricks workspace service principle token expire calculation. Issue : when I am creating a token I have given lifetime =3600, but when I doing get token I am getting unexpected expiry number and even when I ...
- 3355 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi Team, Please help on my issue,Is there any way to find expiry of token, i mean still how much time have token to expiry. creation_time - expiry_time is not giving me exact output.Kindly let me know if there is any way to find as soon as possibleT...
- 1 kudos
- 7030 Views
- 3 replies
- 1 kudos
Resolved! Run databricks job instantly without waiting job cluster get active
when we run databricks job it take some time to get job cluster active . I created pool also and attached with job cluster but still it take time to attached the cluster and job cluster get active to start the job run. is there any way - we can run d...
- 7030 Views
- 3 replies
- 1 kudos
- 1 kudos
If you want instant processing, you will have to have a cluster running all the time.As mentioned above, Databricks is testing serverless compute for data engineering workloads (comparable to serverless SQL). This fires up a cluster in a few seconds...
- 1 kudos
- 6637 Views
- 7 replies
- 8 kudos
Can we run pandas dataframe inside databricks?
Hi, I want to run df=pd.read_csv('/dbfs/FileStore/airlines1.csv') while trying to run getting error likeFileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/airlines1.csv'Could you please help me out how to run pandas dataframe in...
- 6637 Views
- 7 replies
- 8 kudos
- 8 kudos
Hi @Tinendra Kumar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
- 8 kudos
- 7417 Views
- 2 replies
- 6 kudos
Resolved! Can anyone help me to understand one question in PracticeExam-DataEngineerAssociate?
It is the practice exam for data engineer associateThe question is:A data engineering team has created a series of tables using Parquet data stored in an external system. The team is noticing that after appending new rows to the data in the external ...
- 7417 Views
- 2 replies
- 6 kudos
- 6 kudos
Not an answer, just asking the databricks folks to clarify:I would also like to understand this. If there is no event emitted from the external parquet table (push) , and no active pulling or refreshing from the delta table side (pull), how is the un...
- 6 kudos
- 10318 Views
- 4 replies
- 6 kudos
Disable dbutils.fs.put() write to console "Wrote x bytes"
Hey all, does anyone know how to suppress the output of dbutils.fs.put() ?
- 10318 Views
- 4 replies
- 6 kudos
- 6 kudos
Hi @Jordan Fox​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 6 kudos
- 6065 Views
- 5 replies
- 5 kudos
Databricks doesn't stop compute resources in GCP
I started using Databricks in Google Cloud but it charges some unexpected costs. When I create a cluster I notice some compute resources being created in GCP but when I stop the cluster these resources are still up and never shut down. This issue res...
- 6065 Views
- 5 replies
- 5 kudos
- 5 kudos
The answer to the question about the kubernetes cluster regardless of dbx compute and dwh resources running is provided in this thread: https://community.databricks.com/s/question/0D58Y00009TbWqtSAF/auto-termination-for-clusters-jobs-and-delta-live-t...
- 5 kudos
- 5790 Views
- 6 replies
- 2 kudos
Setting up my first DLT Pipeline with 3rd party JSON data
I'm getting an error when I try to create a DLT Pipeline from a bunch of third-party app-usage data we have. Here's the error message:Found invalid character(s) among ' ,;{}()\n\t=' in the column names of your schema. Please upgrade your Delta table ...
- 5790 Views
- 6 replies
- 2 kudos
- 2 kudos
I found this other forum thread that looks potentially useful, but I can’t figure out either how to translate it to SQL to handle JSON, nor how to get the pipeline I’m working with to interpret the Python. When I switch to Python, it complains about ...
- 2 kudos
- 5476 Views
- 4 replies
- 0 kudos
What determines whether an experiment's "Rename/Permissions/Delete" context menu is active or grayed out in the Experiments page?
I have a couple of experiments in the machine learning workspace. Some I want to delete since they are clutter/were created just to test out the platform. However I can't because the options to delete them is grayed out (see pictures below)I was wond...
- 5476 Views
- 4 replies
- 0 kudos
- 0 kudos
It looks like from the documentation the user should have the 'can Manage' permission:https://docs.databricks.com/security/access-control/workspace-acl.html#mlflow-experiment-permissions-1
- 0 kudos
- 10457 Views
- 5 replies
- 4 kudos
Send custom logs to AWS cloudwatch from Notebook
I would like to send some custom logs (in Python) from my Databricks notebook to AWS Cloudwatch. For example: df = spark.read.json(".......................")logger.info("Successfully ingested data from json")Has someone succeeded in doing this before...
- 10457 Views
- 5 replies
- 4 kudos
- 4 kudos
Hi, You can integrate, please refer: https://aws.amazon.com/blogs/mt/how-to-monitor-databricks-with-amazon-cloudwatch/ and also you can configure audit logging to S3 and redirect it to cloudwatch from AWS. , refer: https://aws.amazon.com/blogs/mt/how...
- 4 kudos
- 2311 Views
- 1 replies
- 2 kudos
Resolved! " Please migrate to `databricks_group_role` "
With Databricks Terraform connector version 1.2.0, I use the following to make the AWS instance profile available to all users in the workspace:// Create AWS instance profileresource "aws_iam_instance_profile" "this" { name = "${var.prefix}_instance_...
- 2311 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello, I opened a support case a couple of months ago specifically about this. The answer I got was"Terraform team will revert the deprecated resource and update the document accordingly.", which has not happened so far. Either provide documentation ...
- 2 kudos
- 4207 Views
- 3 replies
- 4 kudos
Resolved! Data size inflates massively while ingesting
GoalImport and consolidate GBs / TBs of local data in 20-mb chunk parquet files into Databricks / Delta lake / partitioned tables.What I've DoneI took a small subset of data, roughly 72.5 GB and ingested using streaming below. The data is already seq...
- 4207 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Erik Louie​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 4 kudos
- 12469 Views
- 5 replies
- 2 kudos
Resolved! Errors on Python API for DeltaTables on Databricks Spark 10.4 LST
Hi,I am using an (Azure) Databricks Compute cluster in a Jupyter notebook using the Databricks connect Python package. Our cluster is on Databrick runtime 10.4 LST and coherently I am using the databricks-connect==10.4.18.In my notebook I am able to ...
- 12469 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Maarten van Raaij​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....
- 2 kudos
- 1852 Views
- 1 replies
- 1 kudos
Resolved! Didn't receive my certification
I took my databrick certification exam on Feb-08-2023 however did'nt receive my certification untill now . Can i get help ?
- 1852 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Djoumoi Moinsalima​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training and our team will get back to you shortly.
- 1 kudos
- 2627 Views
- 1 replies
- 0 kudos
Unable to connect to Azure Cosmos DB Cassandra API table using Azure databricks job
Getting below error:Query [id = , runId = ] terminated with exception: Failed to open native connection to Cassandra at {<name>.cassandra.cosmosdb.azure.com:10350} :: Method com/microsoft/azure/cosmosdb/cassandra/CosmosDbConnectionFactory$.createSess...
- 2627 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, The error looks like there is an issue connecting to csms-ws-ddicsg-dev-001.cassandra.cosmosdb.azure.com:10350. Could you please reverify this in networking config? Also, it will be helpful if you raise an Azure case simultaneously to check the n...
- 0 kudos
- 3761 Views
- 5 replies
- 1 kudos
Resolved! Data Engineer Certificate and badge not received
I have cleared the certification exam on 26th January 2023 with 91.11%, but still haven't received the certificate. I had given the exam with a different mail ID but I'm not receiving any emails from Databricks to that mail ID.​I've also raised a tic...
- 3761 Views
- 5 replies
- 1 kudos
- 1 kudos
Hello team, I have answered via salesforce ticket and sent Naeem their sign on link and resent the badge/certificate email. Thanks!
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 575 | |
| 349 | |
| 287 |
