- 11464 Views
- 2 replies
- 1 kudos
how to use R in databricks
Hello everyone.I am a new user of databricks, they implemented it in the company where I work. I am a business analyst and I know something about R, not much either, when I saw that databricks could use R I was very excited because I thought that the...
- 11464 Views
- 2 replies
- 1 kudos
- 1 kudos
There are some existing posts about using R in databricks:https://docs.gcp.databricks.com/en/sparkr/index.htmlhttps://docs.databricks.com/en/dev-tools/databricks-connect/cluster-config.htmlOnce you have the correct cluster started (this post is about...
- 1 kudos
- 8715 Views
- 8 replies
- 1 kudos
log signature and input data for Spark LinearRegression
I am looking for a way to log my `pyspark.ml.regression.LinearRegression` model with input and signature ata. The usual example that I found around are using sklearn and they can simply do # Log the model with signature and input example signature =...
- 8715 Views
- 8 replies
- 1 kudos
- 1 kudos
I accidentally stumbled upon this ticket when researching on a similar issue. Note that starting from MLflow 2.15.0 it supports VectorUDT. https://mlflow.org/releases/2.15.0
- 1 kudos
- 526 Views
- 0 replies
- 0 kudos
Patient Risk Score based on health history: Unable to create data folder for artifacts in S3 bucket
Hi All,we're using the below git project to build PoC on the concept of "Patient-Level Risk Scoring Based on Condition History": https://github.com/databricks-industry-solutions/hls-patient-riskI was able to import the solution into Databricks and ru...
- 526 Views
- 0 replies
- 0 kudos
- 844 Views
- 1 replies
- 0 kudos
Data Bricks x Power BI Report Server
I connected two .pbix files to the local server. In the first, I used Import connectivity, and in the second, Direct Query connectivity. However, I encountered the following problems: Import connection: The data is viewed successfully, but it is not ...
- 844 Views
- 1 replies
- 0 kudos
- 0 kudos
@Iguinrj11 wrote:I connected two .pbix files to the local server. In the first, I used Import connectivity, and in the second, Direct Query connectivity. However, I encountered the following problems: Import connection: The data is viewed successfull...
- 0 kudos
- 1846 Views
- 5 replies
- 0 kudos
Building a Custom Usage Dashboard using APIs for Job-Level Cost Insights
Since Databricks does not provide individual cost breakdowns for components like Jobs or Compute, we aim to create a custom usage dashboard leveraging APIs to display the cost of each job run across Databricks, Azure Data Factory (ADF), or serverless...
- 1846 Views
- 5 replies
- 0 kudos
- 0 kudos
Hey,Yes, I am not Azure expert but, Databricks REST API can help you extract usage data for serverless resources, allowing you to integrate this information into custom dashboards or external tools like Grafana.On the Azure side, costs related to wil...
- 0 kudos
- 3365 Views
- 4 replies
- 1 kudos
Resolved! Permission denied during write
Hey everyone,I have a pipeline that fetches data from s3 and stores them under the Databricks .tmp/ folder.The pipeline is always able to write around 200 000 files before I get a Permission Denied error. This happens in the following code block: os....
- 3365 Views
- 4 replies
- 1 kudos
- 1 kudos
Thanks for your reply Walter! The filenames are already unique, retries produce the same result and I have the necessary permission as I was able to write the other 200 000 files (with the same program that is running continuous). It does makes sense...
- 1 kudos
- 15809 Views
- 12 replies
- 0 kudos
Resolved! databricks data engineer associate exam
Hello Team, I encountered Pathetic experience while attempting my 1st Databricks certification. I was giving the exam and Abruptly, Proctor asked me to show my desk, everything i showed every corner of my bed.. It was neat and clean with no suspiciou...
- 15809 Views
- 12 replies
- 0 kudos
- 0 kudos
Hi @gokul2 the badge was issued on Dec 2. We just resent the email. Please check your spam. If you continue to have issues, please file a ticket with our support team: https://help.databricks.com/s/contact-us?ReqType=training
- 0 kudos
- 1555 Views
- 3 replies
- 0 kudos
Resolved! How to grant custom container AWS credentials for reading init script?
I'm using a customer container *and* init scripts. At runtime, I get this error:Cluster '...' was terminated. Reason: INIT_SCRIPT_FAILURE (CLIENT_ERROR). Parameters: instance_id:i-0440ddd3a2d5cce79, databricks_error_message:Cluster scoped init script...
- 1555 Views
- 3 replies
- 0 kudos
- 0 kudos
Followup: I got the AWS creds working by amending our AWS role to permit read/write access to our S3 bucket. Woohoo!
- 0 kudos
- 1815 Views
- 3 replies
- 0 kudos
Resolved! Format when specifying docker_image url?
I am providing a custom Docker image to my Databricks/Spark job. I've created the image and uploaded it to our private ECR registry (the URL is `472542229217.dkr.ecr.us-west-2.amazonaws.com/tectonai/mrstevegross-testing:latest`). Based on the docs (h...
- 1815 Views
- 3 replies
- 0 kudos
- 0 kudos
Thanks, that's pretty much what I did; a lot of terraform configuration to get the AWS account set up properly, and now I'm able to tell DBR to load the container. (FWIW, I'm encountering *new* access issues; I started a thread here (https://communit...
- 0 kudos
- 1040 Views
- 1 replies
- 1 kudos
Unable to Register Models After Uploading Artifacts to DBFS in Databricks
Hi everyone,I'm currently working on a project where I'm migrating models and artifacts from a source Databricks workspace to a target one. I've written a script to upload the model artifacts from my local system to DBFS in the target workspace (usi...
- 1040 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Sudheer2, Does it give you any error while trying to register the model?
- 1 kudos
- 891 Views
- 2 replies
- 0 kudos
Scheduling multiple jobs (workflows) in DABs
Hello, I'm wondering how can I schedule multiple jobs (workflow).I'd like to do something like this but on a workflow level. tasks: - task_key: task_1 sql_task: warehouse_id: ${var.warehouse_id} paramet...
- 891 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Greg_c, You can try with this sctructure: - In the main databricks.yml # databricks.ymlbundle:name: master-bundle include:- resources/*.yml # Other bundle configurations... In resource directory, create a YAML for each job: # resources/job1.ymlre...
- 0 kudos
- 2012 Views
- 4 replies
- 1 kudos
ALIAS Not accepted 42601
I am unable to run the following query generated from my backend at databricks sideQuery: SELECT "A".`cut` AS "Cut" , "A".`color` AS "Color" , "A".`carat` AS "Carat" , "A".`clarity` AS "Clarity" FROM databricksconnect.default.diamonds "A" Error logs...
- 2012 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @malhm ,Double quotes are not supported in column alias. In Databricks SQL/Spark SQL one uses backticks instead of double quotes like in PostgreSQL.Check the docs:https://spark.apache.org/docs/3.5.1/sql-ref-identifier.html
- 1 kudos
- 3885 Views
- 3 replies
- 0 kudos
Resolved! DataBricks x Query Folding Power BI
I ran a native Power BI query in DataBricks in import mode and query folding was not enabled. No query folding?
- 3885 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Iguinrj11 ,The trick is to configure Databricks.Query instead of Databricks.Catalogs.Check this article and let us know if that helps:https://www.linkedin.com/pulse/query-folding-azure-databricks-tushar-desai/
- 0 kudos
- 1410 Views
- 2 replies
- 0 kudos
Dynamic Bloom Filters for Inner Joins
I have a question regarding combining the use of Bloom filters with Liquid Clustering to further reduce the data read during a join/merge on top of dynamic file pruning. Testing both combined worked extremely well together for point queries. However ...
- 1410 Views
- 2 replies
- 0 kudos
- 0 kudos
We do not recommend Bloom filters Index on the Delta Tables as they have to be manually maintained. If you prefer photon - please try predictive I/O with Liquid Clustering.
- 0 kudos
- 1677 Views
- 4 replies
- 0 kudos
Automate run as workflow parameter to default to current user
I am trying to run a workflow within Databricks. I have 2 workflows, workflow one which always runs as the service principal, as all data gets accessed and wrangled within this workflow, and workflow 2 which always defaults to the last run account. I...
- 1677 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi, how are you expecting to achieve this? Do you want users who are manually triggering this workflow first update to their run_as? or you want to make this happen programatically?
- 0 kudos
